[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250114 09:09:45.697073 7606 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.109.190:34111
I20250114 09:09:45.697413 7606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 09:09:45.697568 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:45.702211 7614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:45.702666 7612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:45.703328 7606 server_base.cc:1034] running on GCE node
W20250114 09:09:45.704317 7611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:45.704587 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:45.704641 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:45.704665 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845785704665 us; error 0 us; skew 500 ppm
I20250114 09:09:45.708639 7606 webserver.cc:458] Webserver started at http://127.7.109.190:43803/ using document root <none> and password file <none>
I20250114 09:09:45.708820 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:45.708871 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:45.708951 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:45.709991 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/master-0-root/instance:
uuid: "190e64a9f87c43928aac2916fd5a64bb"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:45.711117 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20250114 09:09:45.711654 7619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.711792 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.711858 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/master-0-root
uuid: "190e64a9f87c43928aac2916fd5a64bb"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:45.711913 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:45.726051 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:45.726334 7606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 09:09:45.726447 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:45.726778 7625 maintenance_manager.cc:345] Maintenance manager is disabled (check --enable_maintenance_manager).
I20250114 09:09:45.730335 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.190:34111
I20250114 09:09:45.730388 7671 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.190:34111 every 8 connection(s)
I20250114 09:09:45.730963 7672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:45.733566 7672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb: Bootstrap starting.
I20250114 09:09:45.734315 7672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:45.734545 7672 log.cc:826] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb: Log is configured to *not* fsync() on all Append() calls
I20250114 09:09:45.735034 7672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb: No bootstrap required, opened a new log
I20250114 09:09:45.736343 7672 raft_consensus.cc:357] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190e64a9f87c43928aac2916fd5a64bb" member_type: VOTER }
I20250114 09:09:45.736434 7672 raft_consensus.cc:383] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:45.736459 7672 raft_consensus.cc:738] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190e64a9f87c43928aac2916fd5a64bb, State: Initialized, Role: FOLLOWER
I20250114 09:09:45.736583 7672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190e64a9f87c43928aac2916fd5a64bb" member_type: VOTER }
I20250114 09:09:45.736636 7672 raft_consensus.cc:397] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 09:09:45.736662 7672 raft_consensus.cc:491] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 09:09:45.736704 7672 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:45.737146 7672 raft_consensus.cc:513] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190e64a9f87c43928aac2916fd5a64bb" member_type: VOTER }
I20250114 09:09:45.737237 7672 leader_election.cc:304] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 190e64a9f87c43928aac2916fd5a64bb; no voters:
I20250114 09:09:45.737368 7672 leader_election.cc:290] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 09:09:45.737412 7675 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 1 FOLLOWER]: Leader election won for term 1
I20250114 09:09:45.737608 7672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [sys.catalog]: configured and running, proceeding with master startup.
I20250114 09:09:45.737594 7675 raft_consensus.cc:695] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 1 LEADER]: Becoming Leader. State: Replica: 190e64a9f87c43928aac2916fd5a64bb, State: Running, Role: LEADER
I20250114 09:09:45.737740 7675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190e64a9f87c43928aac2916fd5a64bb" member_type: VOTER }
I20250114 09:09:45.738085 7676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "190e64a9f87c43928aac2916fd5a64bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190e64a9f87c43928aac2916fd5a64bb" member_type: VOTER } }
I20250114 09:09:45.738065 7677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190e64a9f87c43928aac2916fd5a64bb. Latest consensus state: current_term: 1 leader_uuid: "190e64a9f87c43928aac2916fd5a64bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190e64a9f87c43928aac2916fd5a64bb" member_type: VOTER } }
I20250114 09:09:45.738258 7677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:45.738489 7684 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 09:09:45.738246 7676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:45.738976 7684 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 09:09:45.739184 7606 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 09:09:45.740278 7684 catalog_manager.cc:1348] Generated new cluster ID: d7158f90a9b94bb0a593cfc8c404d13f
I20250114 09:09:45.740324 7684 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 09:09:45.748966 7684 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 09:09:45.749637 7684 catalog_manager.cc:1505] Loading token signing keys...
I20250114 09:09:45.753878 7684 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb: Generated new TSK 0
I20250114 09:09:45.754051 7684 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 09:09:45.754839 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:45.757062 7695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:45.757632 7694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:45.758237 7697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:45.759271 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:45.759387 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:45.759413 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:45.759431 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845785759431 us; error 0 us; skew 500 ppm
I20250114 09:09:45.760126 7606 webserver.cc:458] Webserver started at http://127.7.109.129:45867/ using document root <none> and password file <none>
I20250114 09:09:45.760223 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:45.760272 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:45.760316 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:45.760620 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root/instance:
uuid: "451e6007c71a4ba390db946424f2de96"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:45.761467 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:45.761978 7702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.762111 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.762166 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root
uuid: "451e6007c71a4ba390db946424f2de96"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:45.762212 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:45.783443 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:45.783743 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:45.784018 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:45.784359 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:45.784402 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.784440 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:45.784463 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.788427 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.129:36199
I20250114 09:09:45.788483 7765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.129:36199 every 8 connection(s)
I20250114 09:09:45.788847 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250114 09:09:45.791105 7606 server_base.cc:1034] running on GCE node
W20250114 09:09:45.792104 7768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:45.792809 7771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:45.793368 7773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:45.793596 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:45.793639 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:45.793658 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845785793657 us; error 0 us; skew 500 ppm
I20250114 09:09:45.793807 7766 heartbeater.cc:346] Connected to a master server at 127.7.109.190:34111
I20250114 09:09:45.793889 7766 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:45.794086 7766 heartbeater.cc:510] Master 127.7.109.190:34111 requested a full tablet report, sending...
I20250114 09:09:45.794348 7606 webserver.cc:458] Webserver started at http://127.7.109.130:40795/ using document root <none> and password file <none>
I20250114 09:09:45.794435 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:45.794474 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:45.794512 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:45.794595 7636 ts_manager.cc:194] Registered new tserver with Master: 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199)
I20250114 09:09:45.794796 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-1-root/instance:
uuid: "72e3b3eace9d4c608e072268a52e72b0"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:45.795218 7636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56850
I20250114 09:09:45.796020 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:45.796546 7778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.796676 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250114 09:09:45.796727 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-1-root
uuid: "72e3b3eace9d4c608e072268a52e72b0"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:45.796773 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:45.824949 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:45.825238 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:45.825424 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:45.825634 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:45.825666 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.825695 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:45.825719 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.829835 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.130:34959
I20250114 09:09:45.829896 7841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.130:34959 every 8 connection(s)
I20250114 09:09:45.830207 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:45.832212 7846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:45.833329 7842 heartbeater.cc:346] Connected to a master server at 127.7.109.190:34111
I20250114 09:09:45.833393 7842 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:45.833494 7842 heartbeater.cc:510] Master 127.7.109.190:34111 requested a full tablet report, sending...
W20250114 09:09:45.833680 7845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:45.833777 7636 ts_manager.cc:194] Registered new tserver with Master: 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:45.834124 7636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56864
W20250114 09:09:45.834769 7848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:45.835521 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:45.835631 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:45.835659 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:45.835675 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845785835675 us; error 0 us; skew 500 ppm
I20250114 09:09:45.836308 7606 webserver.cc:458] Webserver started at http://127.7.109.131:46483/ using document root <none> and password file <none>
I20250114 09:09:45.836386 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:45.836432 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:45.836478 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:45.836778 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-2-root/instance:
uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:45.837596 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:45.838002 7853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.838109 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.838145 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-2-root
uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:45.838186 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:45.859824 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:45.860141 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:45.860358 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:45.860631 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:45.860673 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.860707 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:45.860733 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:45.865085 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.131:41699
I20250114 09:09:45.865303 7916 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.131:41699 every 8 connection(s)
I20250114 09:09:45.870186 7917 heartbeater.cc:346] Connected to a master server at 127.7.109.190:34111
I20250114 09:09:45.870252 7917 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:45.870344 7917 heartbeater.cc:510] Master 127.7.109.190:34111 requested a full tablet report, sending...
I20250114 09:09:45.870580 7636 ts_manager.cc:194] Registered new tserver with Master: 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699)
I20250114 09:09:45.870721 7606 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005485124s
I20250114 09:09:45.870926 7636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56880
I20250114 09:09:45.879406 7606 test_util.cc:274] Using random seed: -333917334
I20250114 09:09:45.886030 7636 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 09:09:45.886281 7636 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 09:09:45.892308 7806 tablet_service.cc:1467] Processing CreateTablet for tablet fc1dd60afd80432d814ae457e2d29ca4 (DEFAULT_TABLE table=test-workload [id=bdadc6ec0e9a44b983afc9f90e370260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:45.892518 7806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1dd60afd80432d814ae457e2d29ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:45.893409 7730 tablet_service.cc:1467] Processing CreateTablet for tablet fc1dd60afd80432d814ae457e2d29ca4 (DEFAULT_TABLE table=test-workload [id=bdadc6ec0e9a44b983afc9f90e370260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:45.893507 7730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1dd60afd80432d814ae457e2d29ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:45.892308 7881 tablet_service.cc:1467] Processing CreateTablet for tablet fc1dd60afd80432d814ae457e2d29ca4 (DEFAULT_TABLE table=test-workload [id=bdadc6ec0e9a44b983afc9f90e370260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:45.894274 7881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1dd60afd80432d814ae457e2d29ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:45.895453 7940 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc: Bootstrap starting.
I20250114 09:09:45.894145 7938 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Bootstrap starting.
I20250114 09:09:45.895622 7939 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Bootstrap starting.
I20250114 09:09:45.896047 7939 tablet_bootstrap.cc:654] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:45.896786 7938 tablet_bootstrap.cc:654] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:45.897305 7938 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: No bootstrap required, opened a new log
I20250114 09:09:45.897359 7938 ts_tablet_manager.cc:1397] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250114 09:09:45.897505 7938 raft_consensus.cc:357] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:45.897583 7938 raft_consensus.cc:383] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:45.897603 7938 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e3b3eace9d4c608e072268a52e72b0, State: Initialized, Role: FOLLOWER
I20250114 09:09:45.897671 7938 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:45.897846 7938 ts_tablet_manager.cc:1428] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250114 09:09:45.897886 7842 heartbeater.cc:502] Master 127.7.109.190:34111 was elected leader, sending a full tablet report...
I20250114 09:09:45.898267 7940 tablet_bootstrap.cc:654] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:45.898679 7939 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: No bootstrap required, opened a new log
I20250114 09:09:45.898727 7939 ts_tablet_manager.cc:1397] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250114 09:09:45.898950 7940 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc: No bootstrap required, opened a new log
I20250114 09:09:45.899039 7940 ts_tablet_manager.cc:1397] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250114 09:09:45.899215 7939 raft_consensus.cc:357] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:45.899335 7939 raft_consensus.cc:383] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:45.899271 7940 raft_consensus.cc:357] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:45.899402 7939 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Initialized, Role: FOLLOWER
I20250114 09:09:45.899538 7939 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:45.899688 7939 ts_tablet_manager.cc:1428] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:45.899418 7940 raft_consensus.cc:383] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:45.899727 7766 heartbeater.cc:502] Master 127.7.109.190:34111 was elected leader, sending a full tablet report...
I20250114 09:09:45.899749 7940 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd3538cfb3c4df08a5b08d26afff4dc, State: Initialized, Role: FOLLOWER
I20250114 09:09:45.899816 7940 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:45.900015 7917 heartbeater.cc:502] Master 127.7.109.190:34111 was elected leader, sending a full tablet report...
I20250114 09:09:45.900004 7940 ts_tablet_manager.cc:1428] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:45.923991 7946 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 09:09:45.924093 7946 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:45.924335 7946 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:45.927222 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0" is_pre_election: true
I20250114 09:09:45.927191 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96" is_pre_election: true
I20250114 09:09:45.927337 7816 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fd3538cfb3c4df08a5b08d26afff4dc in term 0.
I20250114 09:09:45.927337 7740 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fd3538cfb3c4df08a5b08d26afff4dc in term 0.
I20250114 09:09:45.927500 7855 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96; no voters:
I20250114 09:09:45.927580 7946 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 09:09:45.927613 7946 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:45.927630 7946 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:45.928217 7946 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:45.928339 7946 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 1 election: Requested vote from peers 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:45.928479 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:45.928580 7816 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:45.928476 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:45.928629 7740 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:45.929061 7816 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fd3538cfb3c4df08a5b08d26afff4dc in term 1.
I20250114 09:09:45.929199 7854 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 72e3b3eace9d4c608e072268a52e72b0; no voters:
I20250114 09:09:45.929266 7946 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 1 FOLLOWER]: Leader election won for term 1
I20250114 09:09:45.929308 7740 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fd3538cfb3c4df08a5b08d26afff4dc in term 1.
I20250114 09:09:45.929383 7946 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 1 LEADER]: Becoming Leader. State: Replica: 2fd3538cfb3c4df08a5b08d26afff4dc, State: Running, Role: LEADER
I20250114 09:09:45.929430 7946 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:45.930078 7635 catalog_manager.cc:5526] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131). New cstate: current_term: 1 leader_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:45.937822 7816 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 1 FOLLOWER]: Refusing update from remote peer 2fd3538cfb3c4df08a5b08d26afff4dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 09:09:45.937848 7740 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 1 FOLLOWER]: Refusing update from remote peer 2fd3538cfb3c4df08a5b08d26afff4dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 09:09:45.938088 7956 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 09:09:45.938170 7946 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 09:09:45.954520 7606 test_util.cc:274] Using random seed: -333842218
I20250114 09:09:45.959707 7606 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250114 09:09:46.253912 7945 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2fd3538cfb3c4df08a5b08d26afff4dc)
I20250114 09:09:46.254009 7945 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 1 FOLLOWER]: Advancing to term 2
I20250114 09:09:46.254709 7945 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.254907 7945 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 2 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:46.257314 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:46.257359 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:46.257581 7703 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451e6007c71a4ba390db946424f2de96; no voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:46.257661 7945 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 09:09:46.269943 7740 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 2 FOLLOWER]: Rejecting Update request from peer 2fd3538cfb3c4df08a5b08d26afff4dc for earlier term 1. Current term is 2. Ops: []
I20250114 09:09:46.270284 7983 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 }, Status: INVALID_TERM, Last received: 1.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
W20250114 09:09:46.270474 7948 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 09:09:46.270522 7948 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250114 09:09:46.276672 7948 raft_consensus.cc:3049] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 1 LEADER]: Stepping down as leader of term 1
I20250114 09:09:46.276758 7948 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2fd3538cfb3c4df08a5b08d26afff4dc, State: Running, Role: LEADER
I20250114 09:09:46.276842 7948 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 43, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.277000 7948 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 1 FOLLOWER]: Advancing to term 2
I20250114 09:09:46.277683 7943 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2fd3538cfb3c4df08a5b08d26afff4dc)
I20250114 09:09:46.277740 7943 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 1 FOLLOWER]: Advancing to term 2
I20250114 09:09:46.278340 7943 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.278522 7943 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 2 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199)
I20250114 09:09:46.280827 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:46.280834 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:46.280977 7740 raft_consensus.cc:2388] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 72e3b3eace9d4c608e072268a52e72b0 in current term 2: Already voted for candidate 451e6007c71a4ba390db946424f2de96 in this term.
I20250114 09:09:46.281461 7891 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e3b3eace9d4c608e072268a52e72b0 in term 2.
I20250114 09:09:46.281621 7782 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 72e3b3eace9d4c608e072268a52e72b0; no voters: 451e6007c71a4ba390db946424f2de96
I20250114 09:09:46.281715 7943 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 09:09:46.281806 7943 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 2 LEADER]: Becoming Leader. State: Replica: 72e3b3eace9d4c608e072268a52e72b0, State: Running, Role: LEADER
I20250114 09:09:46.281847 7943 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 43, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.282312 7636 catalog_manager.cc:5526] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 reported cstate change: term changed from 1 to 2, leader changed from 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131) to 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130). New cstate: current_term: 2 leader_uuid: "72e3b3eace9d4c608e072268a52e72b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:46.367198 7945 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:46.367300 7945 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 2 FOLLOWER]: Advancing to term 3
I20250114 09:09:46.367957 7945 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.368134 7945 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 3 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:46.368345 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 3 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:46.368414 7891 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 2 FOLLOWER]: Advancing to term 3
I20250114 09:09:46.368391 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 3 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:46.369009 7891 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451e6007c71a4ba390db946424f2de96 in term 3.
I20250114 09:09:46.369195 7706 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96; no voters: 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:46.369316 7945 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 09:09:46.369452 7945 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 3 LEADER]: Becoming Leader. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Running, Role: LEADER
I20250114 09:09:46.369505 7945 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 43, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.369948 7636 catalog_manager.cc:5526] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 reported cstate change: term changed from 2 to 3, leader changed from 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130) to 451e6007c71a4ba390db946424f2de96 (127.7.109.129). New cstate: current_term: 3 leader_uuid: "451e6007c71a4ba390db946424f2de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:46.405970 7891 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 3 FOLLOWER]: Rejecting Update request from peer 72e3b3eace9d4c608e072268a52e72b0 for earlier term 2. Current term is 3. Ops: []
I20250114 09:09:46.406306 7740 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 3 LEADER]: Rejecting Update request from peer 72e3b3eace9d4c608e072268a52e72b0 for earlier term 2. Current term is 3. Ops: []
I20250114 09:09:46.406302 7943 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 }, Status: INVALID_TERM, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 09:09:46.406538 7986 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 }, Status: INVALID_TERM, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 09:09:46.406597 7986 consensus_queue.cc:1225] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Peer 451e6007c71a4ba390db946424f2de96 log is divergent from this leader: its last log entry 3.44 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 43
I20250114 09:09:46.426564 7943 raft_consensus.cc:3049] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 09:09:46.426669 7943 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 72e3b3eace9d4c608e072268a52e72b0, State: Running, Role: LEADER
I20250114 09:09:46.426743 7943 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 2.44, Last appended by leader: 44, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.426872 7943 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 2 FOLLOWER]: Advancing to term 3
I20250114 09:09:46.463502 7816 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 3 FOLLOWER]: Refusing update from remote peer 451e6007c71a4ba390db946424f2de96: Log matching property violated. Preceding OpId in replica: term: 2 index: 44. Preceding OpId from leader: term: 3 index: 44. (term mismatch)
I20250114 09:09:46.463585 7816 pending_rounds.cc:77] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Aborting all ops after (but not including) 43
I20250114 09:09:46.463621 7816 pending_rounds.cc:99] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Aborting uncommitted NO_OP operation due to leader change: 2.44
I20250114 09:09:46.463652 7816 raft_consensus.cc:2883] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 09:09:46.463861 7945 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 09:09:46.480398 7891 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 3 FOLLOWER]: Refusing update from remote peer 451e6007c71a4ba390db946424f2de96: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 3 index: 44. (index mismatch)
I20250114 09:09:46.480666 7945 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 09:09:46.666788 7948 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 3 FOLLOWER]: Starting leader election (detected failure of leader 451e6007c71a4ba390db946424f2de96)
I20250114 09:09:46.666893 7948 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 3 FOLLOWER]: Advancing to term 4
I20250114 09:09:46.667543 7948 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.667699 7948 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 4 election: Requested vote from peers 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:46.667927 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:46.667940 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:46.668205 7854 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc; no voters: 451e6007c71a4ba390db946424f2de96, 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:46.668294 7948 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 09:09:46.687284 7891 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 4 FOLLOWER]: Rejecting Update request from peer 451e6007c71a4ba390db946424f2de96 for earlier term 3. Current term is 4. Ops: []
I20250114 09:09:46.687567 7988 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 }, Status: INVALID_TERM, Last received: 3.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250114 09:09:46.739789 7988 raft_consensus.cc:3049] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 09:09:46.739886 7988 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Running, Role: LEADER
I20250114 09:09:46.739961 7988 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 3.44, Last appended by leader: 44, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.740085 7988 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 3 FOLLOWER]: Advancing to term 4
I20250114 09:09:46.782575 7943 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 451e6007c71a4ba390db946424f2de96)
I20250114 09:09:46.782677 7943 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 3 FOLLOWER]: Advancing to term 4
I20250114 09:09:46.783357 7943 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.783546 7943 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 4 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199)
I20250114 09:09:46.783762 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:46.783766 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:46.783882 7891 raft_consensus.cc:2388] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 72e3b3eace9d4c608e072268a52e72b0 in current term 4: Already voted for candidate 2fd3538cfb3c4df08a5b08d26afff4dc in this term.
I20250114 09:09:46.784593 7740 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e3b3eace9d4c608e072268a52e72b0 in term 4.
I20250114 09:09:46.784770 7780 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451e6007c71a4ba390db946424f2de96, 72e3b3eace9d4c608e072268a52e72b0; no voters: 2fd3538cfb3c4df08a5b08d26afff4dc
I20250114 09:09:46.784865 7943 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 09:09:46.784938 7943 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 4 LEADER]: Becoming Leader. State: Replica: 72e3b3eace9d4c608e072268a52e72b0, State: Running, Role: LEADER
I20250114 09:09:46.784976 7943 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 3.44, Last appended by leader: 44, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.785421 7636 catalog_manager.cc:5526] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 reported cstate change: term changed from 3 to 4, leader changed from 451e6007c71a4ba390db946424f2de96 (127.7.109.129) to 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130). New cstate: current_term: 4 leader_uuid: "72e3b3eace9d4c608e072268a52e72b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:46.803680 7948 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:46.803776 7948 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 4 FOLLOWER]: Advancing to term 5
I20250114 09:09:46.804435 7948 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.804638 7948 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 5 election: Requested vote from peers 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:46.804843 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 5 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:46.804847 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 5 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:46.805025 7740 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 4 FOLLOWER]: Advancing to term 5
I20250114 09:09:46.805577 7740 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fd3538cfb3c4df08a5b08d26afff4dc in term 5.
I20250114 09:09:46.805760 7855 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96; no voters: 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:46.805850 7948 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 5 FOLLOWER]: Leader election won for term 5
I20250114 09:09:46.805956 7948 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 5 LEADER]: Becoming Leader. State: Replica: 2fd3538cfb3c4df08a5b08d26afff4dc, State: Running, Role: LEADER
I20250114 09:09:46.806003 7948 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 3.44, Last appended by leader: 44, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.877184 7634 catalog_manager.cc:5526] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc reported cstate change: term changed from 4 to 5, leader changed from 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130) to 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131). New cstate: current_term: 5 leader_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:46.889600 7740 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 5 FOLLOWER]: Rejecting Update request from peer 72e3b3eace9d4c608e072268a52e72b0 for earlier term 4. Current term is 5. Ops: []
I20250114 09:09:46.889863 7943 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 }, Status: INVALID_TERM, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250114 09:09:46.894784 7891 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 5 LEADER]: Rejecting Update request from peer 72e3b3eace9d4c608e072268a52e72b0 for earlier term 4. Current term is 5. Ops: []
I20250114 09:09:46.895033 7986 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 }, Status: INVALID_TERM, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250114 09:09:46.895118 7986 consensus_queue.cc:1225] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Peer 2fd3538cfb3c4df08a5b08d26afff4dc log is divergent from this leader: its last log entry 5.45 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 44
I20250114 09:09:46.900686 7816 raft_consensus.cc:3049] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 09:09:46.900741 7816 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 72e3b3eace9d4c608e072268a52e72b0, State: Running, Role: LEADER
I20250114 09:09:46.900791 7816 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 4.45, Last appended by leader: 45, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:46.900885 7816 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 4 FOLLOWER]: Advancing to term 5
I20250114 09:09:46.901549 7816 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 5 FOLLOWER]: Refusing update from remote peer 2fd3538cfb3c4df08a5b08d26afff4dc: Log matching property violated. Preceding OpId in replica: term: 4 index: 45. Preceding OpId from leader: term: 5 index: 45. (term mismatch)
I20250114 09:09:46.901587 7816 pending_rounds.cc:77] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Aborting all ops after (but not including) 44
I20250114 09:09:46.901599 7816 pending_rounds.cc:99] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Aborting uncommitted NO_OP operation due to leader change: 4.45
I20250114 09:09:46.901619 7816 raft_consensus.cc:2883] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 09:09:46.901824 7948 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250114 09:09:46.908846 7740 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 5 FOLLOWER]: Refusing update from remote peer 2fd3538cfb3c4df08a5b08d26afff4dc: Log matching property violated. Preceding OpId in replica: term: 3 index: 44. Preceding OpId from leader: term: 5 index: 45. (index mismatch)
I20250114 09:09:46.909121 7948 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250114 09:09:47.204469 7943 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2fd3538cfb3c4df08a5b08d26afff4dc)
I20250114 09:09:47.204600 7943 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 5 FOLLOWER]: Advancing to term 6
I20250114 09:09:47.205250 7943 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.205421 7943 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 6 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199)
I20250114 09:09:47.205636 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:47.205636 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:47.205888 7782 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72e3b3eace9d4c608e072268a52e72b0; no voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96
I20250114 09:09:47.205987 7943 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 09:09:47.210764 7816 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 6 FOLLOWER]: Rejecting Update request from peer 2fd3538cfb3c4df08a5b08d26afff4dc for earlier term 5. Current term is 6. Ops: []
I20250114 09:09:47.210999 7989 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 }, Status: INVALID_TERM, Last received: 5.45, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250114 09:09:47.236234 7990 raft_consensus.cc:3049] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 5 LEADER]: Stepping down as leader of term 5
I20250114 09:09:47.236320 7990 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2fd3538cfb3c4df08a5b08d26afff4dc, State: Running, Role: LEADER
I20250114 09:09:47.236393 7990 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 5.45, Last appended by leader: 45, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.236536 7990 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 5 FOLLOWER]: Advancing to term 6
I20250114 09:09:47.261230 7988 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2fd3538cfb3c4df08a5b08d26afff4dc)
I20250114 09:09:47.261322 7988 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 5 FOLLOWER]: Advancing to term 6
I20250114 09:09:47.261986 7988 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.262166 7988 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 6 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:47.262367 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:47.262370 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:47.262480 7816 raft_consensus.cc:2388] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 451e6007c71a4ba390db946424f2de96 in current term 6: Already voted for candidate 72e3b3eace9d4c608e072268a52e72b0 in this term.
I20250114 09:09:47.262961 7891 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451e6007c71a4ba390db946424f2de96 in term 6.
I20250114 09:09:47.263152 7706 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96; no voters: 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:47.263267 7988 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 09:09:47.263370 7988 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 6 LEADER]: Becoming Leader. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Running, Role: LEADER
I20250114 09:09:47.263422 7988 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 5.45, Last appended by leader: 45, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.263860 7635 catalog_manager.cc:5526] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 reported cstate change: term changed from 5 to 6, leader changed from 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131) to 451e6007c71a4ba390db946424f2de96 (127.7.109.129). New cstate: current_term: 6 leader_uuid: "451e6007c71a4ba390db946424f2de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:47.317852 7943 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:47.317962 7943 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 6 FOLLOWER]: Advancing to term 7
I20250114 09:09:47.318575 7943 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.318724 7943 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 7 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199)
I20250114 09:09:47.318938 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:47.318949 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:47.319039 7891 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 6 FOLLOWER]: Advancing to term 7
I20250114 09:09:47.319622 7891 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e3b3eace9d4c608e072268a52e72b0 in term 7.
I20250114 09:09:47.319780 7782 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 72e3b3eace9d4c608e072268a52e72b0; no voters: 451e6007c71a4ba390db946424f2de96
I20250114 09:09:47.319864 7943 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 09:09:47.319931 7943 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 7 LEADER]: Becoming Leader. State: Replica: 72e3b3eace9d4c608e072268a52e72b0, State: Running, Role: LEADER
I20250114 09:09:47.319979 7943 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 5.45, Last appended by leader: 45, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.323585 7636 catalog_manager.cc:5526] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 reported cstate change: term changed from 6 to 7, leader changed from 451e6007c71a4ba390db946424f2de96 (127.7.109.129) to 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130). New cstate: current_term: 7 leader_uuid: "72e3b3eace9d4c608e072268a52e72b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:47.350703 7891 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 7 FOLLOWER]: Rejecting Update request from peer 451e6007c71a4ba390db946424f2de96 for earlier term 6. Current term is 7. Ops: []
I20250114 09:09:47.350970 7988 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 }, Status: INVALID_TERM, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250114 09:09:47.362169 7988 raft_consensus.cc:3049] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 09:09:47.362262 7988 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Running, Role: LEADER
I20250114 09:09:47.362332 7988 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 6.46, Last appended by leader: 46, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.362432 7988 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 6 FOLLOWER]: Advancing to term 7
I20250114 09:09:47.397919 7740 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 7 FOLLOWER]: Refusing update from remote peer 72e3b3eace9d4c608e072268a52e72b0: Log matching property violated. Preceding OpId in replica: term: 6 index: 46. Preceding OpId from leader: term: 7 index: 46. (term mismatch)
I20250114 09:09:47.398017 7740 pending_rounds.cc:77] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Aborting all ops after (but not including) 45
I20250114 09:09:47.398036 7740 pending_rounds.cc:99] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Aborting uncommitted NO_OP operation due to leader change: 6.46
I20250114 09:09:47.398054 7740 raft_consensus.cc:2883] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 09:09:47.398334 7943 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250114 09:09:47.430904 7891 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 7 FOLLOWER]: Refusing update from remote peer 72e3b3eace9d4c608e072268a52e72b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 45. Preceding OpId from leader: term: 7 index: 46. (index mismatch)
I20250114 09:09:47.431170 7943 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250114 09:09:47.532107 7990 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 7 FOLLOWER]: Starting leader election (detected failure of leader 72e3b3eace9d4c608e072268a52e72b0)
I20250114 09:09:47.532232 7990 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 7 FOLLOWER]: Advancing to term 8
I20250114 09:09:47.533010 7990 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.533217 7990 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 8 election: Requested vote from peers 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:47.533430 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 8 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:47.533430 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 8 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:47.533725 7854 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc; no voters: 451e6007c71a4ba390db946424f2de96, 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:47.533851 7990 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 09:09:47.549242 7891 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 8 FOLLOWER]: Rejecting Update request from peer 72e3b3eace9d4c608e072268a52e72b0 for earlier term 7. Current term is 8. Ops: []
I20250114 09:09:47.549487 7992 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 }, Status: INVALID_TERM, Last received: 7.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250114 09:09:47.634259 7990 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:47.634366 7990 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 8 FOLLOWER]: Advancing to term 9
I20250114 09:09:47.635025 7990 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.635197 7990 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 9 election: Requested vote from peers 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:47.635396 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 9 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:47.635396 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 9 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:47.635620 7855 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc; no voters: 451e6007c71a4ba390db946424f2de96, 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:47.635725 7990 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 09:09:47.638849 7992 raft_consensus.cc:3049] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 09:09:47.638929 7992 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 72e3b3eace9d4c608e072268a52e72b0, State: Running, Role: LEADER
I20250114 09:09:47.639001 7992 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 7.46, Last appended by leader: 46, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.639093 7992 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 7 FOLLOWER]: Advancing to term 8
I20250114 09:09:47.678901 7988 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 72e3b3eace9d4c608e072268a52e72b0)
I20250114 09:09:47.678992 7988 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 7 FOLLOWER]: Advancing to term 8
I20250114 09:09:47.679592 7988 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.679754 7988 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 8 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:47.680004 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 8 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:47.680006 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 8 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:47.680135 7891 raft_consensus.cc:2363] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 451e6007c71a4ba390db946424f2de96 for earlier term 8. Current term is 9.
I20250114 09:09:47.680310 7706 leader_election.cc:400] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 8 election: Vote denied by peer 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699) with higher term. Message: Invalid argument: T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 451e6007c71a4ba390db946424f2de96 for earlier term 8. Current term is 9.
I20250114 09:09:47.680337 7706 leader_election.cc:403] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20250114 09:09:47.680388 7988 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 8 FOLLOWER]: Advancing to term 9
I20250114 09:09:47.680799 7816 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451e6007c71a4ba390db946424f2de96 in term 8.
I20250114 09:09:47.680917 7988 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699) with higher term. Message: Invalid argument: T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 451e6007c71a4ba390db946424f2de96 for earlier term 8. Current term is 9.
I20250114 09:09:47.783330 7988 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:47.783421 7988 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 9 FOLLOWER]: Advancing to term 10
I20250114 09:09:47.784045 7988 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.784214 7988 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 10 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:47.784415 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 10 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:47.784480 7891 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 9 FOLLOWER]: Advancing to term 10
I20250114 09:09:47.784420 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 10 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:47.784566 7816 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 8 FOLLOWER]: Advancing to term 10
I20250114 09:09:47.785077 7891 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451e6007c71a4ba390db946424f2de96 in term 10.
I20250114 09:09:47.785143 7816 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451e6007c71a4ba390db946424f2de96 in term 10.
I20250114 09:09:47.785271 7706 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96; no voters:
I20250114 09:09:47.785349 7988 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 10 FOLLOWER]: Leader election won for term 10
I20250114 09:09:47.785392 7988 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 10 LEADER]: Becoming Leader. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Running, Role: LEADER
I20250114 09:09:47.785439 7988 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 7.46, Last appended by leader: 46, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:47.785889 7635 catalog_manager.cc:5526] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 reported cstate change: term changed from 7 to 10, leader changed from 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130) to 451e6007c71a4ba390db946424f2de96 (127.7.109.129). New cstate: current_term: 10 leader_uuid: "451e6007c71a4ba390db946424f2de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:47.868340 7891 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 10 FOLLOWER]: Refusing update from remote peer 451e6007c71a4ba390db946424f2de96: Log matching property violated. Preceding OpId in replica: term: 7 index: 46. Preceding OpId from leader: term: 10 index: 47. (index mismatch)
I20250114 09:09:47.868594 7988 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250114 09:09:47.900463 7816 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 10 FOLLOWER]: Refusing update from remote peer 451e6007c71a4ba390db946424f2de96: Log matching property violated. Preceding OpId in replica: term: 7 index: 46. Preceding OpId from leader: term: 10 index: 47. (index mismatch)
I20250114 09:09:47.900758 7988 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250114 09:09:48.076714 7990 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 10 FOLLOWER]: Starting leader election (detected failure of leader 451e6007c71a4ba390db946424f2de96)
I20250114 09:09:48.076812 7990 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 10 FOLLOWER]: Advancing to term 11
I20250114 09:09:48.077482 7990 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.077647 7990 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 11 election: Requested vote from peers 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:48.077883 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 11 candidate_status { last_received { term: 10 index: 47 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:48.077883 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 11 candidate_status { last_received { term: 10 index: 47 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:48.078197 7854 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc; no voters: 451e6007c71a4ba390db946424f2de96, 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:48.078315 7990 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250114 09:09:48.086333 7992 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 451e6007c71a4ba390db946424f2de96)
I20250114 09:09:48.086417 7992 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 10 FOLLOWER]: Advancing to term 11
I20250114 09:09:48.087077 7992 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.087249 7992 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 11 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199)
I20250114 09:09:48.087463 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 11 candidate_status { last_received { term: 10 index: 47 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:48.087463 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 11 candidate_status { last_received { term: 10 index: 47 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:48.087589 7891 raft_consensus.cc:2388] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 72e3b3eace9d4c608e072268a52e72b0 in current term 11: Already voted for candidate 2fd3538cfb3c4df08a5b08d26afff4dc in this term.
I20250114 09:09:48.087745 7782 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72e3b3eace9d4c608e072268a52e72b0; no voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96
I20250114 09:09:48.087845 7992 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250114 09:09:48.095261 7891 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 11 FOLLOWER]: Rejecting Update request from peer 451e6007c71a4ba390db946424f2de96 for earlier term 10. Current term is 11. Ops: []
I20250114 09:09:48.095506 7993 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 }, Status: INVALID_TERM, Last received: 10.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250114 09:09:48.096230 7816 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 11 FOLLOWER]: Rejecting Update request from peer 451e6007c71a4ba390db946424f2de96 for earlier term 10. Current term is 11. Ops: []
I20250114 09:09:48.096484 7993 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 }, Status: INVALID_TERM, Last received: 10.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250114 09:09:48.146741 7988 raft_consensus.cc:3049] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 10 LEADER]: Stepping down as leader of term 10
I20250114 09:09:48.146842 7988 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Running, Role: LEADER
I20250114 09:09:48.146925 7988 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 10.47, Last appended by leader: 47, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.147055 7988 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 10 FOLLOWER]: Advancing to term 11
I20250114 09:09:48.150918 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:48.152181 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:48.154389 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:48.158583 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:48.166790 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:48.183039 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:48.202729 7990 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:48.202844 7990 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 11 FOLLOWER]: Advancing to term 12
I20250114 09:09:48.203516 7990 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.203694 7990 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 12 election: Requested vote from peers 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:48.203922 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 12 candidate_status { last_received { term: 10 index: 47 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:48.203924 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" candidate_term: 12 candidate_status { last_received { term: 10 index: 47 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:48.204025 7740 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 11 FOLLOWER]: Advancing to term 12
I20250114 09:09:48.204066 7816 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 11 FOLLOWER]: Advancing to term 12
I20250114 09:09:48.204619 7816 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fd3538cfb3c4df08a5b08d26afff4dc in term 12.
I20250114 09:09:48.204640 7740 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fd3538cfb3c4df08a5b08d26afff4dc in term 12.
I20250114 09:09:48.204823 7855 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96, 72e3b3eace9d4c608e072268a52e72b0; no voters:
I20250114 09:09:48.204939 7990 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 12 FOLLOWER]: Leader election won for term 12
I20250114 09:09:48.205070 7990 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 12 LEADER]: Becoming Leader. State: Replica: 2fd3538cfb3c4df08a5b08d26afff4dc, State: Running, Role: LEADER
I20250114 09:09:48.205137 7990 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 10.47, Last appended by leader: 47, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.205619 7635 catalog_manager.cc:5526] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc reported cstate change: term changed from 10 to 12, leader changed from 451e6007c71a4ba390db946424f2de96 (127.7.109.129) to 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131). New cstate: current_term: 12 leader_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:48.215282 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:48.279556 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:48.286057 7816 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 12 FOLLOWER]: Refusing update from remote peer 2fd3538cfb3c4df08a5b08d26afff4dc: Log matching property violated. Preceding OpId in replica: term: 10 index: 47. Preceding OpId from leader: term: 12 index: 48. (index mismatch)
I20250114 09:09:48.286370 7990 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250114 09:09:48.287484 7740 raft_consensus.cc:1270] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 12 FOLLOWER]: Refusing update from remote peer 2fd3538cfb3c4df08a5b08d26afff4dc: Log matching property violated. Preceding OpId in replica: term: 10 index: 47. Preceding OpId from leader: term: 12 index: 48. (index mismatch)
I20250114 09:09:48.287717 7994 consensus_queue.cc:1035] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250114 09:09:48.387863 7992 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2fd3538cfb3c4df08a5b08d26afff4dc)
I20250114 09:09:48.387964 7992 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 12 FOLLOWER]: Advancing to term 13
I20250114 09:09:48.393620 7992 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.393853 7992 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 13 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199)
I20250114 09:09:48.394081 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 13 candidate_status { last_received { term: 12 index: 48 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:48.394079 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 13 candidate_status { last_received { term: 12 index: 48 } } ignore_live_leader: false dest_uuid: "451e6007c71a4ba390db946424f2de96"
I20250114 09:09:48.394381 7782 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72e3b3eace9d4c608e072268a52e72b0; no voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96
I20250114 09:09:48.394515 7992 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20250114 09:09:48.396322 7816 raft_consensus.cc:1235] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 13 FOLLOWER]: Rejecting Update request from peer 2fd3538cfb3c4df08a5b08d26afff4dc for earlier term 12. Current term is 13. Ops: []
I20250114 09:09:48.396544 7995 consensus_queue.cc:1046] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 }, Status: INVALID_TERM, Last received: 12.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250114 09:09:48.396654 7990 raft_consensus.cc:3049] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 12 LEADER]: Stepping down as leader of term 12
I20250114 09:09:48.396697 7990 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2fd3538cfb3c4df08a5b08d26afff4dc, State: Running, Role: LEADER
I20250114 09:09:48.396749 7990 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 12.48, Last appended by leader: 48, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.396836 7990 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 12 FOLLOWER]: Advancing to term 13
I20250114 09:09:48.407822 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:48.407960 7606 log_reader.cc:385] T fc1dd60afd80432d814ae457e2d29ca4: removed 1 log segments from log reader
I20250114 09:09:48.407997 7606 log.cc:1078] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Deleting log segment in path: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root/wals/fc1dd60afd80432d814ae457e2d29ca4/wal-000000001 (ops 1-43)
I20250114 09:09:48.408443 7606 timestamp_advancement-itest.cc:268] Shutting down cluster...
I20250114 09:09:48.408551 7606 master.cc:537] Master@127.7.109.190:34111 shutting down...
I20250114 09:09:48.476615 7991 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2fd3538cfb3c4df08a5b08d26afff4dc)
I20250114 09:09:48.476716 7991 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 12 FOLLOWER]: Advancing to term 13
I20250114 09:09:48.477478 7991 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.477653 7991 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 13 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
I20250114 09:09:48.477869 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 13 candidate_status { last_received { term: 12 index: 48 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
I20250114 09:09:48.477859 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "451e6007c71a4ba390db946424f2de96" candidate_term: 13 candidate_status { last_received { term: 12 index: 48 } } ignore_live_leader: false dest_uuid: "72e3b3eace9d4c608e072268a52e72b0"
I20250114 09:09:48.478066 7816 raft_consensus.cc:2388] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 451e6007c71a4ba390db946424f2de96 in current term 13: Already voted for candidate 72e3b3eace9d4c608e072268a52e72b0 in this term.
I20250114 09:09:48.478462 7606 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 1 LEADER]: Raft consensus shutting down.
I20250114 09:09:48.478540 7606 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:48.478583 7606 tablet_replica.cc:331] T 00000000000000000000000000000000 P 190e64a9f87c43928aac2916fd5a64bb: stopping tablet replica
I20250114 09:09:48.478780 7891 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451e6007c71a4ba390db946424f2de96 in term 13.
I20250114 09:09:48.478932 7706 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 451e6007c71a4ba390db946424f2de96; no voters: 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:48.479063 7991 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 FOLLOWER]: Leader election won for term 13
I20250114 09:09:48.479146 7991 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 LEADER]: Becoming Leader. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Running, Role: LEADER
I20250114 09:09:48.479202 7991 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 12.48, Last appended by leader: 48, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
W20250114 09:09:48.479753 7766 heartbeater.cc:643] Failed to heartbeat to 127.7.109.190:34111 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Service unavailable: service kudu.master.MasterService not registered on Master
I20250114 09:09:48.490981 7606 master.cc:559] Master@127.7.109.190:34111 shutdown complete.
I20250114 09:09:48.492445 7606 tablet_server.cc:178] TabletServer@127.7.109.129:0 shutting down...
I20250114 09:09:48.526013 7992 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:48.526105 7992 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 13 FOLLOWER]: Advancing to term 14
I20250114 09:09:48.526746 7992 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.526921 7992 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 14 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199)
I20250114 09:09:48.527134 7891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1dd60afd80432d814ae457e2d29ca4" candidate_uuid: "72e3b3eace9d4c608e072268a52e72b0" candidate_term: 14 candidate_status { last_received { term: 12 index: 48 } } ignore_live_leader: false dest_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc"
W20250114 09:09:48.527175 7780 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250114 09:09:48.527237 7891 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 13 FOLLOWER]: Advancing to term 14
W20250114 09:09:48.527567 7780 leader_election.cc:336] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 451e6007c71a4ba390db946424f2de96 (127.7.109.129:36199): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250114 09:09:48.527853 7891 raft_consensus.cc:2463] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e3b3eace9d4c608e072268a52e72b0 in term 14.
I20250114 09:09:48.528044 7782 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 72e3b3eace9d4c608e072268a52e72b0; no voters: 451e6007c71a4ba390db946424f2de96
I20250114 09:09:48.528137 7992 raft_consensus.cc:2798] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 14 FOLLOWER]: Leader election won for term 14
I20250114 09:09:48.528225 7992 raft_consensus.cc:695] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 14 LEADER]: Becoming Leader. State: Replica: 72e3b3eace9d4c608e072268a52e72b0, State: Running, Role: LEADER
I20250114 09:09:48.528288 7992 consensus_queue.cc:237] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 12.48, Last appended by leader: 48, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.539906 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:48.540074 7606 tablet_replica.cc:331] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: stopping tablet replica
I20250114 09:09:48.540150 7606 raft_consensus.cc:2238] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 LEADER]: Raft consensus shutting down.
I20250114 09:09:48.540205 7606 pending_rounds.cc:62] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Trying to abort 1 pending ops.
I20250114 09:09:48.540223 7606 pending_rounds.cc:69] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Aborting op as it isn't in flight: id { term: 13 index: 49 } timestamp: 7114120349611335680 op_type: NO_OP noop_request { }
I20250114 09:09:48.540282 7606 raft_consensus.cc:2883] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 09:09:48.540298 7606 raft_consensus.cc:2267] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:48.552095 7606 tablet_server.cc:195] TabletServer@127.7.109.129:0 shutdown complete.
I20250114 09:09:48.553122 7606 tablet_server.cc:178] TabletServer@127.7.109.130:0 shutting down...
I20250114 09:09:48.581445 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:48.581574 7606 tablet_replica.cc:331] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: stopping tablet replica
I20250114 09:09:48.581629 7606 raft_consensus.cc:2238] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 14 LEADER]: Raft consensus shutting down.
I20250114 09:09:48.581703 7606 pending_rounds.cc:62] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Trying to abort 1 pending ops.
I20250114 09:09:48.581728 7606 pending_rounds.cc:69] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0: Aborting op as it isn't in flight: id { term: 14 index: 49 } timestamp: 7114120349812334592 op_type: NO_OP noop_request { }
I20250114 09:09:48.581791 7606 raft_consensus.cc:2883] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 14 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 09:09:48.581821 7606 raft_consensus.cc:2267] T fc1dd60afd80432d814ae457e2d29ca4 P 72e3b3eace9d4c608e072268a52e72b0 [term 14 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:48.593479 7606 tablet_server.cc:195] TabletServer@127.7.109.130:0 shutdown complete.
I20250114 09:09:48.594401 7606 tablet_server.cc:178] TabletServer@127.7.109.131:0 shutting down...
I20250114 09:09:48.617470 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:48.617658 7606 tablet_replica.cc:331] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc: stopping tablet replica
I20250114 09:09:48.617702 7606 raft_consensus.cc:2238] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 14 FOLLOWER]: Raft consensus shutting down.
I20250114 09:09:48.617750 7606 raft_consensus.cc:2267] T fc1dd60afd80432d814ae457e2d29ca4 P 2fd3538cfb3c4df08a5b08d26afff4dc [term 14 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:48.629266 7606 tablet_server.cc:195] TabletServer@127.7.109.131:0 shutdown complete.
I20250114 09:09:48.630096 7606 timestamp_advancement-itest.cc:276] Restarting single tablet server...
I20250114 09:09:48.630213 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:48.632114 7999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:48.632790 8002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:48.633853 8000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:48.634538 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:48.634626 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:48.634654 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:48.634666 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845788634666 us; error 0 us; skew 500 ppm
I20250114 09:09:48.635260 7606 webserver.cc:458] Webserver started at http://127.7.109.129:45867/ using document root <none> and password file <none>
I20250114 09:09:48.635339 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:48.635365 7606 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 09:09:48.635936 7606 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:48.637184 8009 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:48.637316 7606 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:48.637369 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root
uuid: "451e6007c71a4ba390db946424f2de96"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:48.637414 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root/data
Total live blocks: 11
Total live bytes: 10719
Total live bytes (after alignment): 53248
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:48.655683 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:48.655859 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:48.655994 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:48.656282 8016 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250114 09:09:48.657079 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 09:09:48.657126 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:48.657150 7606 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250114 09:09:48.657539 7606 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 09:09:48.657567 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:48.657586 8016 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Bootstrap starting.
I20250114 09:09:48.659135 8016 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=1 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250114 09:09:48.659387 8016 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Bootstrap complete.
I20250114 09:09:48.659503 8016 ts_tablet_manager.cc:1397] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250114 09:09:48.659632 8016 raft_consensus.cc:357] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.659716 8016 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Initialized, Role: FOLLOWER
I20250114 09:09:48.659780 8016 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 48, Last appended: 13.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.659958 8016 ts_tablet_manager.cc:1428] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250114 09:09:48.660578 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.129:36199
I20250114 09:09:48.663625 8075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.129:36199 every 8 connection(s)
W20250114 09:09:48.664709 8076 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250114 09:09:48.668753 7606 timestamp_advancement-itest.cc:268] Shutting down cluster...
I20250114 09:09:48.668823 7606 tablet_server.cc:178] TabletServer@127.7.109.129:36199 shutting down...
I20250114 09:09:48.783288 8063 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:48.783399 8063 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 13 FOLLOWER]: Advancing to term 14
I20250114 09:09:48.784065 8063 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.784274 8063 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 14 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
W20250114 09:09:48.784955 8013 leader_election.cc:336] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699): Network error: Client connection negotiation failed: client connection to 127.7.109.131:41699: connect: Connection refused (error 111)
W20250114 09:09:48.785025 8010 leader_election.cc:336] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959): Network error: Client connection negotiation failed: client connection to 127.7.109.130:34959: connect: Connection refused (error 111)
I20250114 09:09:48.785070 8010 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451e6007c71a4ba390db946424f2de96; no voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:48.785153 8063 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250114 09:09:48.897668 8063 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:48.897766 8063 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 14 FOLLOWER]: Advancing to term 15
I20250114 09:09:48.898422 8063 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.898588 8063 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 15 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
W20250114 09:09:48.899130 8010 leader_election.cc:336] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959): Network error: Client connection negotiation failed: client connection to 127.7.109.130:34959: connect: Connection refused (error 111)
W20250114 09:09:48.899204 8013 leader_election.cc:336] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699): Network error: Client connection negotiation failed: client connection to 127.7.109.131:41699: connect: Connection refused (error 111)
I20250114 09:09:48.899245 8013 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451e6007c71a4ba390db946424f2de96; no voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:48.899327 8063 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20250114 09:09:48.913985 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:48.914139 7606 tablet_replica.cc:331] stopping tablet replica
I20250114 09:09:48.914206 7606 raft_consensus.cc:2238] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 15 FOLLOWER]: Raft consensus shutting down.
I20250114 09:09:48.914259 7606 pending_rounds.cc:62] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Trying to abort 1 pending ops.
I20250114 09:09:48.914278 7606 pending_rounds.cc:69] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Aborting op as it isn't in flight: id { term: 13 index: 49 } timestamp: 7114120349611335680 op_type: NO_OP noop_request { }
I20250114 09:09:48.914327 7606 raft_consensus.cc:2883] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 09:09:48.914350 7606 raft_consensus.cc:2267] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 15 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:48.926174 7606 tablet_server.cc:195] TabletServer@127.7.109.129:36199 shutdown complete.
I20250114 09:09:48.927377 7606 timestamp_advancement-itest.cc:276] Restarting single tablet server...
I20250114 09:09:48.927496 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:48.929482 8089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:48.930168 8087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:48.930552 8086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:48.931667 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:48.931758 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:48.931789 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:48.931802 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845788931803 us; error 0 us; skew 500 ppm
I20250114 09:09:48.932415 7606 webserver.cc:458] Webserver started at http://127.7.109.129:45867/ using document root <none> and password file <none>
I20250114 09:09:48.932482 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:48.932547 7606 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 09:09:48.933104 7606 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:48.934262 8097 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:48.934363 7606 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:48.934408 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root
uuid: "451e6007c71a4ba390db946424f2de96"
format_stamp: "Formatted at 2025-01-14 09:09:45 on dist-test-slave-msv5"
I20250114 09:09:48.934459 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0/minicluster-data/ts-0-root/data
Total live blocks: 11
Total live bytes: 10719
Total live bytes (after alignment): 53248
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:48.941385 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:48.941543 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:48.941663 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:48.942395 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 09:09:48.942425 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:48.942811 7606 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 09:09:48.942837 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:48.942847 8104 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Bootstrap starting.
I20250114 09:09:48.944047 8104 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=1 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250114 09:09:48.944231 8104 tablet_bootstrap.cc:492] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Bootstrap complete.
I20250114 09:09:48.944308 8104 ts_tablet_manager.cc:1397] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:48.944424 8104 raft_consensus.cc:357] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 15 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.944521 8104 raft_consensus.cc:738] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 15 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451e6007c71a4ba390db946424f2de96, State: Initialized, Role: FOLLOWER
I20250114 09:09:48.944579 8104 consensus_queue.cc:260] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 48, Last appended: 13.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:48.944733 8104 ts_tablet_manager.cc:1428] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250114 09:09:48.945735 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.129:36199
I20250114 09:09:48.946133 8163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.129:36199 every 8 connection(s)
W20250114 09:09:48.946873 8164 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:356: Failure
Value of: s.IsCorruption()
Actual: false
Expected: true
OK
I20250114 09:09:48.951129 7606 tablet_server.cc:178] TabletServer@127.7.109.129:36199 shutting down...
I20250114 09:09:49.038311 8141 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:49.038414 8141 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 15 FOLLOWER]: Advancing to term 16
I20250114 09:09:49.039119 8141 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:49.039333 8141 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 16 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
W20250114 09:09:49.039923 8098 leader_election.cc:336] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959): Network error: Client connection negotiation failed: client connection to 127.7.109.130:34959: connect: Connection refused (error 111)
W20250114 09:09:49.040007 8101 leader_election.cc:336] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699): Network error: Client connection negotiation failed: client connection to 127.7.109.131:41699: connect: Connection refused (error 111)
I20250114 09:09:49.040068 8101 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451e6007c71a4ba390db946424f2de96; no voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:49.040153 8141 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20250114 09:09:49.143646 8141 raft_consensus.cc:491] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:49.143754 8141 raft_consensus.cc:3054] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 16 FOLLOWER]: Advancing to term 17
I20250114 09:09:49.144438 8141 raft_consensus.cc:513] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd3538cfb3c4df08a5b08d26afff4dc" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41699 } } peers { permanent_uuid: "451e6007c71a4ba390db946424f2de96" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 36199 } } peers { permanent_uuid: "72e3b3eace9d4c608e072268a52e72b0" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 34959 } }
I20250114 09:09:49.144641 8141 leader_election.cc:290] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 17 election: Requested vote from peers 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699), 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959)
W20250114 09:09:49.145197 8101 leader_election.cc:336] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 2fd3538cfb3c4df08a5b08d26afff4dc (127.7.109.131:41699): Network error: Client connection negotiation failed: client connection to 127.7.109.131:41699: connect: Connection refused (error 111)
W20250114 09:09:49.145267 8098 leader_election.cc:336] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 72e3b3eace9d4c608e072268a52e72b0 (127.7.109.130:34959): Network error: Client connection negotiation failed: client connection to 127.7.109.130:34959: connect: Connection refused (error 111)
I20250114 09:09:49.145308 8098 leader_election.cc:304] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451e6007c71a4ba390db946424f2de96; no voters: 2fd3538cfb3c4df08a5b08d26afff4dc, 72e3b3eace9d4c608e072268a52e72b0
I20250114 09:09:49.145388 8141 raft_consensus.cc:2743] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20250114 09:09:49.196468 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:49.196666 7606 tablet_replica.cc:331] stopping tablet replica
I20250114 09:09:49.196733 7606 raft_consensus.cc:2238] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 17 FOLLOWER]: Raft consensus shutting down.
I20250114 09:09:49.196779 7606 pending_rounds.cc:62] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Trying to abort 1 pending ops.
I20250114 09:09:49.196802 7606 pending_rounds.cc:69] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96: Aborting op as it isn't in flight: id { term: 13 index: 49 } timestamp: 7114120349611335680 op_type: NO_OP noop_request { }
I20250114 09:09:49.196858 7606 raft_consensus.cc:2883] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 09:09:49.196882 7606 raft_consensus.cc:2267] T fc1dd60afd80432d814ae457e2d29ca4 P 451e6007c71a4ba390db946424f2de96 [term 17 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:49.208461 7606 tablet_server.cc:195] TabletServer@127.7.109.129:36199 shutdown complete.
I20250114 09:09:49.210436 7606 test_util.cc:181] -----------------------------------------------
I20250114 09:09:49.210486 7606 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736845785692342-7606-0
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (3514 ms)
[ RUN ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20250114 09:09:49.210964 7606 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.109.190:37941
I20250114 09:09:49.211107 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:49.212893 8175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:49.213572 8176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:49.214591 8178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:49.215315 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:49.215430 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:49.215458 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:49.215476 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845789215475 us; error 0 us; skew 500 ppm
I20250114 09:09:49.216084 7606 webserver.cc:458] Webserver started at http://127.7.109.190:37141/ using document root <none> and password file <none>
I20250114 09:09:49.216161 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:49.216215 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:49.216261 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:49.216578 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/master-0-root/instance:
uuid: "4e1b5921a672413d944ea89ba20d50f5"
format_stamp: "Formatted at 2025-01-14 09:09:49 on dist-test-slave-msv5"
I20250114 09:09:49.217422 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250114 09:09:49.217872 8183 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.217995 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.218039 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/master-0-root
uuid: "4e1b5921a672413d944ea89ba20d50f5"
format_stamp: "Formatted at 2025-01-14 09:09:49 on dist-test-slave-msv5"
I20250114 09:09:49.218084 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:49.230477 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:49.230671 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:49.233114 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.190:37941
I20250114 09:09:49.233572 8235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.190:37941 every 8 connection(s)
I20250114 09:09:49.233695 8236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:49.234655 8236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5: Bootstrap starting.
I20250114 09:09:49.234931 8236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:49.235369 8236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5: No bootstrap required, opened a new log
I20250114 09:09:49.235472 8236 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b5921a672413d944ea89ba20d50f5" member_type: VOTER }
I20250114 09:09:49.235529 8236 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:49.235543 8236 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1b5921a672413d944ea89ba20d50f5, State: Initialized, Role: FOLLOWER
I20250114 09:09:49.235580 8236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b5921a672413d944ea89ba20d50f5" member_type: VOTER }
I20250114 09:09:49.235616 8236 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 09:09:49.235642 8236 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 09:09:49.235661 8236 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:49.236096 8236 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b5921a672413d944ea89ba20d50f5" member_type: VOTER }
I20250114 09:09:49.236150 8236 leader_election.cc:304] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e1b5921a672413d944ea89ba20d50f5; no voters:
I20250114 09:09:49.236245 8236 leader_election.cc:290] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 09:09:49.236336 8239 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 09:09:49.236367 8236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 09:09:49.236481 8239 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e1b5921a672413d944ea89ba20d50f5, State: Running, Role: LEADER
I20250114 09:09:49.236584 8239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b5921a672413d944ea89ba20d50f5" member_type: VOTER }
I20250114 09:09:49.236797 8240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e1b5921a672413d944ea89ba20d50f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b5921a672413d944ea89ba20d50f5" member_type: VOTER } }
I20250114 09:09:49.236917 8240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:49.236819 8241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e1b5921a672413d944ea89ba20d50f5. Latest consensus state: current_term: 1 leader_uuid: "4e1b5921a672413d944ea89ba20d50f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1b5921a672413d944ea89ba20d50f5" member_type: VOTER } }
I20250114 09:09:49.237063 8241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:49.237231 8247 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 09:09:49.237401 8247 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 09:09:49.237794 7606 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 09:09:49.237985 8247 catalog_manager.cc:1348] Generated new cluster ID: 006f14cff5cf4a7d8fcd8e005aa97da4
I20250114 09:09:49.238034 8247 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 09:09:49.268097 8247 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 09:09:49.268527 8247 catalog_manager.cc:1505] Loading token signing keys...
I20250114 09:09:49.277534 8247 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5: Generated new TSK 0
I20250114 09:09:49.277626 8247 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 09:09:49.301694 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:49.303918 8258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:49.304638 8257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:49.305333 8260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:49.305949 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:49.306059 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:49.306087 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:49.306103 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845789306104 us; error 0 us; skew 500 ppm
I20250114 09:09:49.306747 7606 webserver.cc:458] Webserver started at http://127.7.109.129:46679/ using document root <none> and password file <none>
I20250114 09:09:49.306831 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:49.306877 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:49.306926 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:49.307245 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root/instance:
uuid: "656e818763eb45d89ca10460f8d7a3b5"
format_stamp: "Formatted at 2025-01-14 09:09:49 on dist-test-slave-msv5"
I20250114 09:09:49.308036 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250114 09:09:49.308444 8265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.308588 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.308629 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root
uuid: "656e818763eb45d89ca10460f8d7a3b5"
format_stamp: "Formatted at 2025-01-14 09:09:49 on dist-test-slave-msv5"
I20250114 09:09:49.308674 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:49.324725 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:49.324993 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:49.325166 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:49.325403 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:49.325436 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.325469 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:49.325496 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.328403 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.129:42225
I20250114 09:09:49.328435 8328 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.129:42225 every 8 connection(s)
I20250114 09:09:49.328827 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:49.330626 8334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:49.331439 8333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:49.331688 8336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:49.332072 8329 heartbeater.cc:346] Connected to a master server at 127.7.109.190:37941
I20250114 09:09:49.332125 8329 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:49.332229 8329 heartbeater.cc:510] Master 127.7.109.190:37941 requested a full tablet report, sending...
I20250114 09:09:49.332474 8200 ts_manager.cc:194] Registered new tserver with Master: 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:49.332674 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:49.332821 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:49.332852 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:49.332863 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845789332864 us; error 0 us; skew 500 ppm
I20250114 09:09:49.333151 8200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47926
I20250114 09:09:49.333601 7606 webserver.cc:458] Webserver started at http://127.7.109.130:41441/ using document root <none> and password file <none>
I20250114 09:09:49.333681 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:49.333732 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:49.333783 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:49.334090 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-1-root/instance:
uuid: "1d17b287ff3546cebcd685f9965d063f"
format_stamp: "Formatted at 2025-01-14 09:09:49 on dist-test-slave-msv5"
I20250114 09:09:49.334978 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:49.335449 8341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.335587 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.335633 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-1-root
uuid: "1d17b287ff3546cebcd685f9965d063f"
format_stamp: "Formatted at 2025-01-14 09:09:49 on dist-test-slave-msv5"
I20250114 09:09:49.335682 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:49.341275 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:49.341531 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:49.341693 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:49.341884 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:49.341915 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.341948 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:49.341974 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.345297 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.130:39703
I20250114 09:09:49.345352 8404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.130:39703 every 8 connection(s)
I20250114 09:09:49.345685 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:49.347505 8409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:49.348026 8408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:49.348572 8405 heartbeater.cc:346] Connected to a master server at 127.7.109.190:37941
I20250114 09:09:49.348621 8405 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:49.348716 8405 heartbeater.cc:510] Master 127.7.109.190:37941 requested a full tablet report, sending...
I20250114 09:09:49.348940 8200 ts_manager.cc:194] Registered new tserver with Master: 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703)
W20250114 09:09:49.349002 8411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:49.349315 8200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47942
I20250114 09:09:49.349644 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:49.349768 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:49.349804 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:49.349817 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845789349818 us; error 0 us; skew 500 ppm
I20250114 09:09:49.350392 7606 webserver.cc:458] Webserver started at http://127.7.109.131:43445/ using document root <none> and password file <none>
I20250114 09:09:49.350455 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:49.350490 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:49.350522 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:49.350745 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-2-root/instance:
uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
format_stamp: "Formatted at 2025-01-14 09:09:49 on dist-test-slave-msv5"
I20250114 09:09:49.351517 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250114 09:09:49.351907 8416 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.352007 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.352046 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-2-root
uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
format_stamp: "Formatted at 2025-01-14 09:09:49 on dist-test-slave-msv5"
I20250114 09:09:49.352092 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:49.376860 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:49.377130 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:49.377300 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:49.377496 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:49.377522 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.377550 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:49.377570 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.380800 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.131:34755
I20250114 09:09:49.380867 8479 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.131:34755 every 8 connection(s)
I20250114 09:09:49.383733 8480 heartbeater.cc:346] Connected to a master server at 127.7.109.190:37941
I20250114 09:09:49.383802 8480 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:49.383908 8480 heartbeater.cc:510] Master 127.7.109.190:37941 requested a full tablet report, sending...
I20250114 09:09:49.384153 8200 ts_manager.cc:194] Registered new tserver with Master: 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755)
I20250114 09:09:49.384274 7606 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003301518s
I20250114 09:09:49.384639 8200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47946
I20250114 09:09:49.391093 7606 test_util.cc:274] Using random seed: -330405646
I20250114 09:09:49.395262 8200 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 09:09:49.395401 8200 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 09:09:49.399883 8369 tablet_service.cc:1467] Processing CreateTablet for tablet b19f15b21e194615a83631957faebe9a (DEFAULT_TABLE table=test-workload [id=64e0467159214b75a443049c9303c077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:49.400031 8369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b19f15b21e194615a83631957faebe9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:49.400316 8293 tablet_service.cc:1467] Processing CreateTablet for tablet b19f15b21e194615a83631957faebe9a (DEFAULT_TABLE table=test-workload [id=64e0467159214b75a443049c9303c077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:49.400446 8293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b19f15b21e194615a83631957faebe9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:49.400770 8444 tablet_service.cc:1467] Processing CreateTablet for tablet b19f15b21e194615a83631957faebe9a (DEFAULT_TABLE table=test-workload [id=64e0467159214b75a443049c9303c077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:49.400864 8444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b19f15b21e194615a83631957faebe9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:49.401640 8502 tablet_bootstrap.cc:492] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Bootstrap starting.
I20250114 09:09:49.401983 8503 tablet_bootstrap.cc:492] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd: Bootstrap starting.
I20250114 09:09:49.402042 8502 tablet_bootstrap.cc:654] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:49.402391 8501 tablet_bootstrap.cc:492] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f: Bootstrap starting.
I20250114 09:09:49.402448 8503 tablet_bootstrap.cc:654] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:49.402746 8501 tablet_bootstrap.cc:654] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:49.402923 8503 tablet_bootstrap.cc:492] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd: No bootstrap required, opened a new log
I20250114 09:09:49.402963 8503 ts_tablet_manager.cc:1397] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:49.403093 8503 raft_consensus.cc:357] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.403141 8502 tablet_bootstrap.cc:492] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: No bootstrap required, opened a new log
I20250114 09:09:49.403162 8503 raft_consensus.cc:383] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:49.403179 8502 ts_tablet_manager.cc:1397] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250114 09:09:49.403182 8503 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a637534fcda4d8d9c4521fcba5e77fd, State: Initialized, Role: FOLLOWER
I20250114 09:09:49.403255 8501 tablet_bootstrap.cc:492] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f: No bootstrap required, opened a new log
I20250114 09:09:49.403340 8503 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.403354 8501 ts_tablet_manager.cc:1397] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:49.403363 8502 raft_consensus.cc:357] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.403491 8502 raft_consensus.cc:383] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:49.403517 8501 raft_consensus.cc:357] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.403551 8502 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656e818763eb45d89ca10460f8d7a3b5, State: Initialized, Role: FOLLOWER
I20250114 09:09:49.403519 8503 ts_tablet_manager.cc:1428] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:49.403554 8480 heartbeater.cc:502] Master 127.7.109.190:37941 was elected leader, sending a full tablet report...
I20250114 09:09:49.403578 8501 raft_consensus.cc:383] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:49.403673 8502 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.403709 8501 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d17b287ff3546cebcd685f9965d063f, State: Initialized, Role: FOLLOWER
I20250114 09:09:49.403759 8501 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.403839 8502 ts_tablet_manager.cc:1428] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:49.403914 8501 ts_tablet_manager.cc:1428] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:49.403930 8329 heartbeater.cc:502] Master 127.7.109.190:37941 was elected leader, sending a full tablet report...
I20250114 09:09:49.404050 8405 heartbeater.cc:502] Master 127.7.109.190:37941 was elected leader, sending a full tablet report...
I20250114 09:09:49.414505 8507 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 09:09:49.414589 8507 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.414758 8507 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:49.417208 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f" is_pre_election: true
I20250114 09:09:49.417308 8379 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a637534fcda4d8d9c4521fcba5e77fd in term 0.
I20250114 09:09:49.417311 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5" is_pre_election: true
I20250114 09:09:49.417387 8303 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a637534fcda4d8d9c4521fcba5e77fd in term 0.
I20250114 09:09:49.417472 8419 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f, 7a637534fcda4d8d9c4521fcba5e77fd; no voters:
I20250114 09:09:49.417553 8507 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 09:09:49.417582 8507 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:49.417599 8507 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:49.418219 8507 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.418339 8507 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 1 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:49.418493 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:49.418558 8303 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:49.419129 8303 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a637534fcda4d8d9c4521fcba5e77fd in term 1.
I20250114 09:09:49.419222 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:49.419281 8379 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:49.419287 8419 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 656e818763eb45d89ca10460f8d7a3b5, 7a637534fcda4d8d9c4521fcba5e77fd; no voters:
I20250114 09:09:49.419384 8507 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 09:09:49.419524 8507 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 1 LEADER]: Becoming Leader. State: Replica: 7a637534fcda4d8d9c4521fcba5e77fd, State: Running, Role: LEADER
I20250114 09:09:49.419626 8507 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.419868 8379 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a637534fcda4d8d9c4521fcba5e77fd in term 1.
I20250114 09:09:49.420197 8198 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131). New cstate: current_term: 1 leader_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:49.428126 8379 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 1 FOLLOWER]: Refusing update from remote peer 7a637534fcda4d8d9c4521fcba5e77fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 09:09:49.428126 8303 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 1 FOLLOWER]: Refusing update from remote peer 7a637534fcda4d8d9c4521fcba5e77fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 09:09:49.428472 8507 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 09:09:49.428597 8511 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 09:09:49.430519 8520 mvcc.cc:204] Tried to move back new op lower bound from 7114120353497157632 to 7114120353463410688. Current Snapshot: MvccSnapshot[applied={T|T < 7114120353496276992}]
I20250114 09:09:49.748679 8509 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a637534fcda4d8d9c4521fcba5e77fd)
I20250114 09:09:49.748778 8509 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 1 FOLLOWER]: Advancing to term 2
I20250114 09:09:49.749502 8509 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.749707 8509 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 2 election: Requested vote from peers 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:49.752926 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:49.752944 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:49.753170 8343 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f; no voters: 656e818763eb45d89ca10460f8d7a3b5, 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:49.753271 8509 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 09:09:49.760905 8379 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 2 FOLLOWER]: Rejecting Update request from peer 7a637534fcda4d8d9c4521fcba5e77fd for earlier term 1. Current term is 2. Ops: []
I20250114 09:09:49.761173 8535 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: INVALID_TERM, Last received: 1.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250114 09:09:49.786397 8511 raft_consensus.cc:3049] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 09:09:49.786479 8511 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a637534fcda4d8d9c4521fcba5e77fd, State: Running, Role: LEADER
I20250114 09:09:49.786551 8511 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.786661 8511 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 1 FOLLOWER]: Advancing to term 2
I20250114 09:09:49.823457 8508 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a637534fcda4d8d9c4521fcba5e77fd)
I20250114 09:09:49.823557 8508 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 09:09:49.824308 8508 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.824560 8508 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 2 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755)
I20250114 09:09:49.827010 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:49.827092 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:49.827159 8379 raft_consensus.cc:2388] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 656e818763eb45d89ca10460f8d7a3b5 in current term 2: Already voted for candidate 1d17b287ff3546cebcd685f9965d063f in this term.
I20250114 09:09:49.827692 8454 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 656e818763eb45d89ca10460f8d7a3b5 in term 2.
I20250114 09:09:49.827858 8267 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 656e818763eb45d89ca10460f8d7a3b5, 7a637534fcda4d8d9c4521fcba5e77fd; no voters: 1d17b287ff3546cebcd685f9965d063f
I20250114 09:09:49.827965 8508 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 09:09:49.828070 8508 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 2 LEADER]: Becoming Leader. State: Replica: 656e818763eb45d89ca10460f8d7a3b5, State: Running, Role: LEADER
I20250114 09:09:49.828130 8508 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.828608 8200 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 reported cstate change: term changed from 1 to 2, leader changed from 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131) to 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129). New cstate: current_term: 2 leader_uuid: "656e818763eb45d89ca10460f8d7a3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:49.888628 8509 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:49.888724 8509 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 2 FOLLOWER]: Advancing to term 3
I20250114 09:09:49.889397 8509 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.889566 8509 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 3 election: Requested vote from peers 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:49.889792 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 3 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:49.889792 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 3 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:49.889922 8454 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 2 FOLLOWER]: Advancing to term 3
I20250114 09:09:49.890532 8454 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d17b287ff3546cebcd685f9965d063f in term 3.
I20250114 09:09:49.890687 8343 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f, 7a637534fcda4d8d9c4521fcba5e77fd; no voters: 656e818763eb45d89ca10460f8d7a3b5
I20250114 09:09:49.890766 8509 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 3 FOLLOWER]: Leader election won for term 3
I20250114 09:09:49.890868 8509 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 3 LEADER]: Becoming Leader. State: Replica: 1d17b287ff3546cebcd685f9965d063f, State: Running, Role: LEADER
I20250114 09:09:49.890909 8509 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.891374 8200 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f reported cstate change: term changed from 2 to 3, leader changed from 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129) to 1d17b287ff3546cebcd685f9965d063f (127.7.109.130). New cstate: current_term: 3 leader_uuid: "1d17b287ff3546cebcd685f9965d063f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:49.933976 8379 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 3 LEADER]: Rejecting Update request from peer 656e818763eb45d89ca10460f8d7a3b5 for earlier term 2. Current term is 3. Ops: []
I20250114 09:09:49.934266 8508 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250114 09:09:49.934406 8508 consensus_queue.cc:1225] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Peer 1d17b287ff3546cebcd685f9965d063f log is divergent from this leader: its last log entry 3.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20250114 09:09:49.948154 8454 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 3 FOLLOWER]: Rejecting Update request from peer 656e818763eb45d89ca10460f8d7a3b5 for earlier term 2. Current term is 3. Ops: []
I20250114 09:09:49.948393 8508 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250114 09:09:49.964533 8538 raft_consensus.cc:3049] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 09:09:49.964623 8538 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 656e818763eb45d89ca10460f8d7a3b5, State: Running, Role: LEADER
I20250114 09:09:49.964694 8538 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.16, Last appended by leader: 16, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:49.964812 8538 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 2 FOLLOWER]: Advancing to term 3
I20250114 09:09:49.993741 8303 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 3 FOLLOWER]: Refusing update from remote peer 1d17b287ff3546cebcd685f9965d063f: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 3 index: 16. (term mismatch)
I20250114 09:09:49.993816 8303 pending_rounds.cc:77] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Aborting all ops after (but not including) 15
I20250114 09:09:49.993830 8303 pending_rounds.cc:99] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Aborting uncommitted NO_OP operation due to leader change: 2.16
I20250114 09:09:49.993845 8303 raft_consensus.cc:2883] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 09:09:49.994043 8509 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Connected to new peer: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250114 09:09:50.000216 8454 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 3 FOLLOWER]: Refusing update from remote peer 1d17b287ff3546cebcd685f9965d063f: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20250114 09:09:50.000427 8539 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250114 09:09:50.190233 8538 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1d17b287ff3546cebcd685f9965d063f)
I20250114 09:09:50.190337 8538 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 3 FOLLOWER]: Advancing to term 4
I20250114 09:09:50.191009 8538 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.191181 8538 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 4 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755)
I20250114 09:09:50.191380 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:50.191380 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:50.191645 8268 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 656e818763eb45d89ca10460f8d7a3b5; no voters: 1d17b287ff3546cebcd685f9965d063f, 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:50.191759 8538 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 09:09:50.208477 8303 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 4 FOLLOWER]: Rejecting Update request from peer 1d17b287ff3546cebcd685f9965d063f for earlier term 3. Current term is 4. Ops: []
I20250114 09:09:50.208735 8539 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: INVALID_TERM, Last received: 3.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250114 09:09:50.230974 8509 raft_consensus.cc:3049] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 3 LEADER]: Stepping down as leader of term 3
I20250114 09:09:50.231084 8509 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1d17b287ff3546cebcd685f9965d063f, State: Running, Role: LEADER
I20250114 09:09:50.231179 8509 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.231312 8509 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 3 FOLLOWER]: Advancing to term 4
I20250114 09:09:50.263871 8511 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1d17b287ff3546cebcd685f9965d063f)
I20250114 09:09:50.263996 8511 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 3 FOLLOWER]: Advancing to term 4
I20250114 09:09:50.264932 8511 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.265125 8511 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 4 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:50.265352 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:50.266136 8379 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a637534fcda4d8d9c4521fcba5e77fd in term 4.
I20250114 09:09:50.266258 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:50.266310 8419 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f, 7a637534fcda4d8d9c4521fcba5e77fd; no voters:
I20250114 09:09:50.266314 8303 raft_consensus.cc:2388] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a637534fcda4d8d9c4521fcba5e77fd in current term 4: Already voted for candidate 656e818763eb45d89ca10460f8d7a3b5 in this term.
I20250114 09:09:50.266443 8511 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 4 FOLLOWER]: Leader election won for term 4
I20250114 09:09:50.266543 8511 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 4 LEADER]: Becoming Leader. State: Replica: 7a637534fcda4d8d9c4521fcba5e77fd, State: Running, Role: LEADER
I20250114 09:09:50.266599 8511 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.267072 8200 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd reported cstate change: term changed from 3 to 4, leader changed from 1d17b287ff3546cebcd685f9965d063f (127.7.109.130) to 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131). New cstate: current_term: 4 leader_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:50.292652 8538 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:50.292740 8538 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 4 FOLLOWER]: Advancing to term 5
I20250114 09:09:50.293398 8538 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.293568 8538 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 5 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755)
I20250114 09:09:50.293757 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:50.293828 8379 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 4 FOLLOWER]: Advancing to term 5
I20250114 09:09:50.293792 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:50.294353 8379 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 656e818763eb45d89ca10460f8d7a3b5 in term 5.
I20250114 09:09:50.294513 8268 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f, 656e818763eb45d89ca10460f8d7a3b5; no voters: 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:50.294590 8538 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 09:09:50.294642 8538 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 5 LEADER]: Becoming Leader. State: Replica: 656e818763eb45d89ca10460f8d7a3b5, State: Running, Role: LEADER
I20250114 09:09:50.294692 8538 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.333853 8197 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 reported cstate change: term changed from 4 to 5, leader changed from 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131) to 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129). New cstate: current_term: 5 leader_uuid: "656e818763eb45d89ca10460f8d7a3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:50.348521 8303 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 5 LEADER]: Rejecting Update request from peer 7a637534fcda4d8d9c4521fcba5e77fd for earlier term 4. Current term is 5. Ops: []
I20250114 09:09:50.348819 8511 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250114 09:09:50.348933 8511 consensus_queue.cc:1225] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Peer 656e818763eb45d89ca10460f8d7a3b5 log is divergent from this leader: its last log entry 5.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20250114 09:09:50.364588 8379 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 5 FOLLOWER]: Rejecting Update request from peer 7a637534fcda4d8d9c4521fcba5e77fd for earlier term 4. Current term is 5. Ops: []
I20250114 09:09:50.364809 8511 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250114 09:09:50.397481 8379 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 5 FOLLOWER]: Refusing update from remote peer 656e818763eb45d89ca10460f8d7a3b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 16. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
I20250114 09:09:50.397720 8538 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250114 09:09:50.410964 8454 raft_consensus.cc:3049] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 4 LEADER]: Stepping down as leader of term 4
I20250114 09:09:50.411029 8454 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a637534fcda4d8d9c4521fcba5e77fd, State: Running, Role: LEADER
I20250114 09:09:50.411080 8454 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.17, Last appended by leader: 17, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.411190 8454 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 4 FOLLOWER]: Advancing to term 5
I20250114 09:09:50.411787 8454 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 5 FOLLOWER]: Refusing update from remote peer 656e818763eb45d89ca10460f8d7a3b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 5 index: 17. (term mismatch)
I20250114 09:09:50.411826 8454 pending_rounds.cc:77] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd: Aborting all ops after (but not including) 16
I20250114 09:09:50.411839 8454 pending_rounds.cc:99] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd: Aborting uncommitted NO_OP operation due to leader change: 4.17
I20250114 09:09:50.411850 8454 raft_consensus.cc:2883] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 09:09:50.412129 8545 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250114 09:09:50.498922 8509 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 656e818763eb45d89ca10460f8d7a3b5)
I20250114 09:09:50.499015 8509 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 5 FOLLOWER]: Advancing to term 6
I20250114 09:09:50.499650 8509 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.499827 8509 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 6 election: Requested vote from peers 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:50.500041 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:50.500053 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:50.500329 8343 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f; no voters: 656e818763eb45d89ca10460f8d7a3b5, 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:50.500418 8509 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 09:09:50.502763 8379 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 6 FOLLOWER]: Rejecting Update request from peer 656e818763eb45d89ca10460f8d7a3b5 for earlier term 5. Current term is 6. Ops: []
I20250114 09:09:50.502959 8545 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: INVALID_TERM, Last received: 5.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250114 09:09:50.525123 8538 raft_consensus.cc:3049] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 09:09:50.525220 8538 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 656e818763eb45d89ca10460f8d7a3b5, State: Running, Role: LEADER
I20250114 09:09:50.525282 8538 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 5.17, Last appended by leader: 17, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.525378 8538 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 5 FOLLOWER]: Advancing to term 6
I20250114 09:09:50.614789 8509 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:50.614884 8509 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 6 FOLLOWER]: Advancing to term 7
I20250114 09:09:50.615509 8509 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.615657 8509 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 7 election: Requested vote from peers 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:50.615864 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 7 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:50.615928 8454 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 5 FOLLOWER]: Advancing to term 7
I20250114 09:09:50.615895 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 7 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:50.615991 8303 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 6 FOLLOWER]: Advancing to term 7
I20250114 09:09:50.616484 8454 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d17b287ff3546cebcd685f9965d063f in term 7.
I20250114 09:09:50.616603 8303 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d17b287ff3546cebcd685f9965d063f in term 7.
I20250114 09:09:50.616710 8343 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f, 7a637534fcda4d8d9c4521fcba5e77fd; no voters:
I20250114 09:09:50.616806 8509 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 7 FOLLOWER]: Leader election won for term 7
I20250114 09:09:50.616868 8509 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 7 LEADER]: Becoming Leader. State: Replica: 1d17b287ff3546cebcd685f9965d063f, State: Running, Role: LEADER
I20250114 09:09:50.616915 8509 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 5.17, Last appended by leader: 17, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.617328 8197 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f reported cstate change: term changed from 5 to 7, leader changed from 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129) to 1d17b287ff3546cebcd685f9965d063f (127.7.109.130). New cstate: current_term: 7 leader_uuid: "1d17b287ff3546cebcd685f9965d063f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:50.721727 8544 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:50.721827 8544 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 7 FOLLOWER]: Advancing to term 8
I20250114 09:09:50.722472 8544 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.722615 8544 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 8 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:50.722824 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 8 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:50.722848 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 8 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:50.722930 8303 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 7 FOLLOWER]: Advancing to term 8
I20250114 09:09:50.723446 8303 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a637534fcda4d8d9c4521fcba5e77fd in term 8.
I20250114 09:09:50.723619 8419 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 656e818763eb45d89ca10460f8d7a3b5, 7a637534fcda4d8d9c4521fcba5e77fd; no voters:
I20250114 09:09:50.723707 8544 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 8 FOLLOWER]: Leader election won for term 8
I20250114 09:09:50.723758 8544 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 8 LEADER]: Becoming Leader. State: Replica: 7a637534fcda4d8d9c4521fcba5e77fd, State: Running, Role: LEADER
I20250114 09:09:50.723809 8544 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 5.17, Last appended by leader: 17, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.724278 8197 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd reported cstate change: term changed from 7 to 8, leader changed from 1d17b287ff3546cebcd685f9965d063f (127.7.109.130) to 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131). New cstate: current_term: 8 leader_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:50.729655 8454 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 8 LEADER]: Rejecting Update request from peer 1d17b287ff3546cebcd685f9965d063f for earlier term 7. Current term is 8. Ops: []
I20250114 09:09:50.729866 8509 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250114 09:09:50.729950 8509 consensus_queue.cc:1225] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Peer 7a637534fcda4d8d9c4521fcba5e77fd log is divergent from this leader: its last log entry 8.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20250114 09:09:50.731727 8303 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 8 FOLLOWER]: Rejecting Update request from peer 1d17b287ff3546cebcd685f9965d063f for earlier term 7. Current term is 8. Ops: []
I20250114 09:09:50.731927 8509 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250114 09:09:50.793087 8539 raft_consensus.cc:3049] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 7 LEADER]: Stepping down as leader of term 7
I20250114 09:09:50.793192 8539 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1d17b287ff3546cebcd685f9965d063f, State: Running, Role: LEADER
I20250114 09:09:50.793268 8539 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 7.18, Last appended by leader: 18, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:50.793372 8539 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 7 FOLLOWER]: Advancing to term 8
I20250114 09:09:50.829878 8303 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 8 FOLLOWER]: Refusing update from remote peer 7a637534fcda4d8d9c4521fcba5e77fd: Log matching property violated. Preceding OpId in replica: term: 5 index: 17. Preceding OpId from leader: term: 8 index: 18. (index mismatch)
I20250114 09:09:50.830132 8544 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250114 09:09:50.838367 8379 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 8 FOLLOWER]: Refusing update from remote peer 7a637534fcda4d8d9c4521fcba5e77fd: Log matching property violated. Preceding OpId in replica: term: 7 index: 18. Preceding OpId from leader: term: 8 index: 18. (term mismatch)
I20250114 09:09:50.838447 8379 pending_rounds.cc:77] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f: Aborting all ops after (but not including) 17
I20250114 09:09:50.838465 8379 pending_rounds.cc:99] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f: Aborting uncommitted NO_OP operation due to leader change: 7.18
I20250114 09:09:50.838485 8379 raft_consensus.cc:2883] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 09:09:50.838675 8544 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250114 09:09:51.013581 8538 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7a637534fcda4d8d9c4521fcba5e77fd)
I20250114 09:09:51.013677 8538 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 8 FOLLOWER]: Advancing to term 9
I20250114 09:09:51.014315 8538 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.014570 8538 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 9 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755)
I20250114 09:09:51.014659 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:51.014833 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:51.015072 8268 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 656e818763eb45d89ca10460f8d7a3b5; no voters: 1d17b287ff3546cebcd685f9965d063f, 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:51.015162 8538 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250114 09:09:51.018208 8303 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 9 FOLLOWER]: Rejecting Update request from peer 7a637534fcda4d8d9c4521fcba5e77fd for earlier term 8. Current term is 9. Ops: []
I20250114 09:09:51.018427 8546 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: INVALID_TERM, Last received: 8.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 09:09:51.026626 8546 raft_consensus.cc:3049] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 8 LEADER]: Stepping down as leader of term 8
I20250114 09:09:51.026695 8546 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7a637534fcda4d8d9c4521fcba5e77fd, State: Running, Role: LEADER
I20250114 09:09:51.026765 8546 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 8.18, Last appended by leader: 18, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.026862 8546 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 8 FOLLOWER]: Advancing to term 9
I20250114 09:09:51.115890 8539 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7a637534fcda4d8d9c4521fcba5e77fd)
I20250114 09:09:51.115985 8539 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 8 FOLLOWER]: Advancing to term 9
I20250114 09:09:51.116549 8538 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:51.116621 8538 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 9 FOLLOWER]: Advancing to term 10
I20250114 09:09:51.116676 8539 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.116854 8539 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 9 election: Requested vote from peers 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:51.117051 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 9 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:51.117051 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 9 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:51.117337 8538 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.117458 8303 raft_consensus.cc:2363] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d17b287ff3546cebcd685f9965d063f for earlier term 9. Current term is 10.
I20250114 09:09:51.117547 8538 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 10 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755)
I20250114 09:09:51.117614 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 10 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:51.117668 8379 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 9 FOLLOWER]: Advancing to term 10
I20250114 09:09:51.117687 8344 leader_election.cc:400] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 9 election: Vote denied by peer 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225) with higher term. Message: Invalid argument: T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d17b287ff3546cebcd685f9965d063f for earlier term 9. Current term is 10.
I20250114 09:09:51.117740 8344 leader_election.cc:403] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20250114 09:09:51.117750 8454 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d17b287ff3546cebcd685f9965d063f in term 9.
I20250114 09:09:51.117798 8453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 10 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:51.117847 8453 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 9 FOLLOWER]: Advancing to term 10
I20250114 09:09:51.118304 8379 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 656e818763eb45d89ca10460f8d7a3b5 in term 10.
I20250114 09:09:51.118392 8539 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225) with higher term. Message: Invalid argument: T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d17b287ff3546cebcd685f9965d063f for earlier term 9. Current term is 10.
I20250114 09:09:51.118407 8453 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 656e818763eb45d89ca10460f8d7a3b5 in term 10.
I20250114 09:09:51.118510 8268 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f, 656e818763eb45d89ca10460f8d7a3b5; no voters:
I20250114 09:09:51.118592 8538 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 10 FOLLOWER]: Leader election won for term 10
I20250114 09:09:51.118691 8538 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 10 LEADER]: Becoming Leader. State: Replica: 656e818763eb45d89ca10460f8d7a3b5, State: Running, Role: LEADER
I20250114 09:09:51.118758 8538 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 8.18, Last appended by leader: 18, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.119213 8197 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 reported cstate change: term changed from 8 to 10, leader changed from 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131) to 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129). New cstate: current_term: 10 leader_uuid: "656e818763eb45d89ca10460f8d7a3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:51.211660 8379 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 10 FOLLOWER]: Refusing update from remote peer 656e818763eb45d89ca10460f8d7a3b5: Log matching property violated. Preceding OpId in replica: term: 8 index: 18. Preceding OpId from leader: term: 10 index: 19. (index mismatch)
I20250114 09:09:51.211927 8538 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 09:09:51.213953 8453 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 10 FOLLOWER]: Refusing update from remote peer 656e818763eb45d89ca10460f8d7a3b5: Log matching property violated. Preceding OpId in replica: term: 8 index: 18. Preceding OpId from leader: term: 10 index: 19. (index mismatch)
I20250114 09:09:51.214183 8548 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 09:09:51.315526 8546 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 10 FOLLOWER]: Starting leader election (detected failure of leader 656e818763eb45d89ca10460f8d7a3b5)
I20250114 09:09:51.315624 8546 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 10 FOLLOWER]: Advancing to term 11
I20250114 09:09:51.316246 8546 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.316387 8546 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 11 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:51.316663 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:51.316663 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:51.316952 8419 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a637534fcda4d8d9c4521fcba5e77fd; no voters: 1d17b287ff3546cebcd685f9965d063f, 656e818763eb45d89ca10460f8d7a3b5
I20250114 09:09:51.317029 8546 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250114 09:09:51.320704 8453 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 11 FOLLOWER]: Rejecting Update request from peer 656e818763eb45d89ca10460f8d7a3b5 for earlier term 10. Current term is 11. Ops: []
I20250114 09:09:51.320957 8547 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 }, Status: INVALID_TERM, Last received: 10.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 09:09:51.397212 8539 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 656e818763eb45d89ca10460f8d7a3b5)
I20250114 09:09:51.397307 8539 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 10 FOLLOWER]: Advancing to term 11
I20250114 09:09:51.397907 8539 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.398053 8539 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 11 election: Requested vote from peers 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:51.398288 8453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:51.398289 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:51.398388 8453 raft_consensus.cc:2388] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d17b287ff3546cebcd685f9965d063f in current term 11: Already voted for candidate 7a637534fcda4d8d9c4521fcba5e77fd in this term.
I20250114 09:09:51.398607 8344 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f; no voters: 656e818763eb45d89ca10460f8d7a3b5, 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:51.398700 8539 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250114 09:09:51.407258 8379 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 11 FOLLOWER]: Rejecting Update request from peer 656e818763eb45d89ca10460f8d7a3b5 for earlier term 10. Current term is 11. Ops: []
I20250114 09:09:51.407521 8547 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: INVALID_TERM, Last received: 10.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 09:09:51.407663 8453 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 11 FOLLOWER]: Rejecting Update request from peer 656e818763eb45d89ca10460f8d7a3b5 for earlier term 10. Current term is 11. Ops: []
I20250114 09:09:51.407838 8538 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 }, Status: INVALID_TERM, Last received: 10.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 09:09:51.417661 8548 raft_consensus.cc:3049] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 10 LEADER]: Stepping down as leader of term 10
I20250114 09:09:51.417749 8548 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 656e818763eb45d89ca10460f8d7a3b5, State: Running, Role: LEADER
I20250114 09:09:51.417835 8548 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 10.19, Last appended by leader: 19, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.417941 8548 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 10 FOLLOWER]: Advancing to term 11
I20250114 09:09:51.419363 8546 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:51.419428 8546 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 11 FOLLOWER]: Advancing to term 12
I20250114 09:09:51.419919 8546 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.420047 8546 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 12 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:51.420267 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 12 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:51.420318 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" candidate_term: 12 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:51.420361 8379 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 11 FOLLOWER]: Advancing to term 12
I20250114 09:09:51.420375 8303 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 11 FOLLOWER]: Advancing to term 12
I20250114 09:09:51.420908 8303 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a637534fcda4d8d9c4521fcba5e77fd in term 12.
I20250114 09:09:51.421028 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.421025 8379 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a637534fcda4d8d9c4521fcba5e77fd in term 12.
I20250114 09:09:51.421152 8419 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 656e818763eb45d89ca10460f8d7a3b5, 7a637534fcda4d8d9c4521fcba5e77fd; no voters:
I20250114 09:09:51.421253 8546 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 12 FOLLOWER]: Leader election won for term 12
I20250114 09:09:51.421309 8546 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 12 LEADER]: Becoming Leader. State: Replica: 7a637534fcda4d8d9c4521fcba5e77fd, State: Running, Role: LEADER
I20250114 09:09:51.421361 8546 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 10.19, Last appended by leader: 19, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.421821 8197 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd reported cstate change: term changed from 10 to 12, leader changed from 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129) to 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131). New cstate: current_term: 12 leader_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:51.422214 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.424366 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.428580 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.436800 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.453044 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.485282 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.511339 8379 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 12 FOLLOWER]: Refusing update from remote peer 7a637534fcda4d8d9c4521fcba5e77fd: Log matching property violated. Preceding OpId in replica: term: 10 index: 19. Preceding OpId from leader: term: 12 index: 20. (index mismatch)
I20250114 09:09:51.511593 8546 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 09:09:51.512575 8303 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 12 FOLLOWER]: Refusing update from remote peer 7a637534fcda4d8d9c4521fcba5e77fd: Log matching property violated. Preceding OpId in replica: term: 10 index: 19. Preceding OpId from leader: term: 12 index: 20. (index mismatch)
I20250114 09:09:51.512869 8546 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 09:09:51.549530 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.613353 8539 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 7a637534fcda4d8d9c4521fcba5e77fd)
I20250114 09:09:51.613467 8539 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 12 FOLLOWER]: Advancing to term 13
I20250114 09:09:51.614130 8539 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.614308 8539 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 13 election: Requested vote from peers 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755), 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225)
I20250114 09:09:51.614547 8453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 13 candidate_status { last_received { term: 12 index: 20 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:51.614549 8303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "1d17b287ff3546cebcd685f9965d063f" candidate_term: 13 candidate_status { last_received { term: 12 index: 20 } } ignore_live_leader: false dest_uuid: "656e818763eb45d89ca10460f8d7a3b5"
I20250114 09:09:51.614670 8303 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 12 FOLLOWER]: Advancing to term 13
I20250114 09:09:51.615237 8303 raft_consensus.cc:2463] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d17b287ff3546cebcd685f9965d063f in term 13.
I20250114 09:09:51.615408 8344 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d17b287ff3546cebcd685f9965d063f, 656e818763eb45d89ca10460f8d7a3b5; no voters: 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:51.615499 8539 raft_consensus.cc:2798] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 13 FOLLOWER]: Leader election won for term 13
I20250114 09:09:51.615613 8539 raft_consensus.cc:695] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 13 LEADER]: Becoming Leader. State: Replica: 1d17b287ff3546cebcd685f9965d063f, State: Running, Role: LEADER
I20250114 09:09:51.615660 8539 consensus_queue.cc:237] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 12.20, Last appended by leader: 20, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.616115 8197 catalog_manager.cc:5526] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f reported cstate change: term changed from 12 to 13, leader changed from 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131) to 1d17b287ff3546cebcd685f9965d063f (127.7.109.130). New cstate: current_term: 13 leader_uuid: "1d17b287ff3546cebcd685f9965d063f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:51.616439 8303 raft_consensus.cc:1235] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 13 FOLLOWER]: Rejecting Update request from peer 7a637534fcda4d8d9c4521fcba5e77fd for earlier term 12. Current term is 13. Ops: []
I20250114 09:09:51.616747 8550 consensus_queue.cc:1046] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: INVALID_TERM, Last received: 12.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 09:09:51.616910 8511 raft_consensus.cc:3049] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 12 LEADER]: Stepping down as leader of term 12
I20250114 09:09:51.616959 8511 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7a637534fcda4d8d9c4521fcba5e77fd, State: Running, Role: LEADER
I20250114 09:09:51.617015 8511 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 12.20, Last appended by leader: 20, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:51.617089 8511 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 12 FOLLOWER]: Advancing to term 13
I20250114 09:09:51.677793 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.709489 8453 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 13 FOLLOWER]: Refusing update from remote peer 1d17b287ff3546cebcd685f9965d063f: Log matching property violated. Preceding OpId in replica: term: 12 index: 20. Preceding OpId from leader: term: 13 index: 21. (index mismatch)
I20250114 09:09:51.709766 8539 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 09:09:51.710789 8303 raft_consensus.cc:1270] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 13 FOLLOWER]: Refusing update from remote peer 1d17b287ff3546cebcd685f9965d063f: Log matching property violated. Preceding OpId in replica: term: 12 index: 20. Preceding OpId from leader: term: 13 index: 21. (index mismatch)
I20250114 09:09:51.711047 8539 consensus_queue.cc:1035] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [LEADER]: Connected to new peer: Peer: permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 09:09:51.934044 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:51.934163 7606 log_reader.cc:385] T b19f15b21e194615a83631957faebe9a: removed 1 log segments from log reader
I20250114 09:09:51.934177 7606 log.cc:1078] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Deleting log segment in path: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root/wals/b19f15b21e194615a83631957faebe9a/wal-000000001 (ops 1-15)
I20250114 09:09:51.934531 7606 timestamp_advancement-itest.cc:268] Shutting down cluster...
I20250114 09:09:51.934590 7606 master.cc:537] Master@127.7.109.190:37941 shutting down...
I20250114 09:09:51.983114 7606 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 1 LEADER]: Raft consensus shutting down.
I20250114 09:09:51.983229 7606 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:51.983253 7606 tablet_replica.cc:331] T 00000000000000000000000000000000 P 4e1b5921a672413d944ea89ba20d50f5: stopping tablet replica
I20250114 09:09:51.995430 7606 master.cc:559] Master@127.7.109.190:37941 shutdown complete.
I20250114 09:09:51.996656 7606 tablet_server.cc:178] TabletServer@127.7.109.129:0 shutting down...
I20250114 09:09:52.074113 8547 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1d17b287ff3546cebcd685f9965d063f)
I20250114 09:09:52.074222 8547 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 13 FOLLOWER]: Advancing to term 14
I20250114 09:09:52.074939 8547 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:52.075119 8547 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 14 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755)
I20250114 09:09:52.075343 8379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 14 candidate_status { last_received { term: 13 index: 21 } } ignore_live_leader: false dest_uuid: "1d17b287ff3546cebcd685f9965d063f"
I20250114 09:09:52.075343 8453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19f15b21e194615a83631957faebe9a" candidate_uuid: "656e818763eb45d89ca10460f8d7a3b5" candidate_term: 14 candidate_status { last_received { term: 13 index: 21 } } ignore_live_leader: false dest_uuid: "7a637534fcda4d8d9c4521fcba5e77fd"
I20250114 09:09:52.075592 8268 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 656e818763eb45d89ca10460f8d7a3b5; no voters: 1d17b287ff3546cebcd685f9965d063f, 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:52.075685 8547 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250114 09:09:52.080225 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:52.080370 7606 tablet_replica.cc:331] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: stopping tablet replica
I20250114 09:09:52.080446 7606 raft_consensus.cc:2238] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 14 FOLLOWER]: Raft consensus shutting down.
I20250114 09:09:52.080490 7606 raft_consensus.cc:2267] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 14 FOLLOWER]: Raft consensus is shut down!
W20250114 09:09:52.084331 8344 consensus_peers.cc:487] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f -> Peer 656e818763eb45d89ca10460f8d7a3b5 (127.7.109.129:42225): Couldn't send request to peer 656e818763eb45d89ca10460f8d7a3b5. Status: Network error: Client connection negotiation failed: client connection to 127.7.109.129:42225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 09:09:52.092360 7606 tablet_server.cc:195] TabletServer@127.7.109.129:0 shutdown complete.
I20250114 09:09:52.093315 7606 tablet_server.cc:178] TabletServer@127.7.109.130:0 shutting down...
I20250114 09:09:52.096865 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:52.096997 7606 tablet_replica.cc:331] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f: stopping tablet replica
I20250114 09:09:52.097054 7606 raft_consensus.cc:2238] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 13 LEADER]: Raft consensus shutting down.
I20250114 09:09:52.097128 7606 raft_consensus.cc:2267] T b19f15b21e194615a83631957faebe9a P 1d17b287ff3546cebcd685f9965d063f [term 13 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:52.098778 7606 tablet_server.cc:195] TabletServer@127.7.109.130:0 shutdown complete.
I20250114 09:09:52.099604 7606 tablet_server.cc:178] TabletServer@127.7.109.131:0 shutting down...
I20250114 09:09:52.132449 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:52.132688 7606 tablet_replica.cc:331] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd: stopping tablet replica
I20250114 09:09:52.132752 7606 raft_consensus.cc:2238] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 13 FOLLOWER]: Raft consensus shutting down.
I20250114 09:09:52.132810 7606 raft_consensus.cc:2267] T b19f15b21e194615a83631957faebe9a P 7a637534fcda4d8d9c4521fcba5e77fd [term 13 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:52.144393 7606 tablet_server.cc:195] TabletServer@127.7.109.131:0 shutdown complete.
I20250114 09:09:52.145262 7606 timestamp_advancement-itest.cc:276] Restarting single tablet server...
I20250114 09:09:52.145382 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:52.147490 8555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:52.148180 8557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:52.148972 8554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:52.149713 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:52.149816 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:52.149843 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:52.149859 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845792149860 us; error 0 us; skew 500 ppm
I20250114 09:09:52.153895 7606 webserver.cc:458] Webserver started at http://127.7.109.129:46679/ using document root <none> and password file <none>
I20250114 09:09:52.153996 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:52.154160 7606 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 09:09:52.154889 7606 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:52.156118 8564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.156260 7606 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:52.156311 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root
uuid: "656e818763eb45d89ca10460f8d7a3b5"
format_stamp: "Formatted at 2025-01-14 09:09:49 on dist-test-slave-msv5"
I20250114 09:09:52.156361 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736845785692342-7606-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5268
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:52.179857 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:52.180042 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:52.180191 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:52.180474 8571 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250114 09:09:52.181056 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 09:09:52.181097 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:52.181119 7606 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250114 09:09:52.181511 7606 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 09:09:52.181541 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.181552 8571 tablet_bootstrap.cc:492] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Bootstrap starting.
I20250114 09:09:52.182760 8571 tablet_bootstrap.cc:492] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=1 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 09:09:52.182966 8571 tablet_bootstrap.cc:492] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Bootstrap complete.
I20250114 09:09:52.183055 8571 ts_tablet_manager.cc:1397] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250114 09:09:52.183182 8571 raft_consensus.cc:357] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 14 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:52.183244 8571 raft_consensus.cc:738] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 14 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656e818763eb45d89ca10460f8d7a3b5, State: Initialized, Role: FOLLOWER
I20250114 09:09:52.183321 8571 consensus_queue.cc:260] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 13.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:52.183480 8571 ts_tablet_manager.cc:1428] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250114 09:09:52.184587 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.129:42225
I20250114 09:09:52.186600 8630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.129:42225 every 8 connection(s)
W20250114 09:09:52.187366 8631 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250114 09:09:52.192493 7606 tablet_server.cc:178] TabletServer@127.7.109.129:42225 shutting down...
I20250114 09:09:52.271888 8608 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:52.271992 8608 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 14 FOLLOWER]: Advancing to term 15
I20250114 09:09:52.272676 8608 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:52.272871 8608 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 15 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755)
W20250114 09:09:52.273510 8566 leader_election.cc:336] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755): Network error: Client connection negotiation failed: client connection to 127.7.109.131:34755: connect: Connection refused (error 111)
W20250114 09:09:52.273594 8567 leader_election.cc:336] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703): Network error: Client connection negotiation failed: client connection to 127.7.109.130:39703: connect: Connection refused (error 111)
I20250114 09:09:52.273664 8567 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 656e818763eb45d89ca10460f8d7a3b5; no voters: 1d17b287ff3546cebcd685f9965d063f, 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:52.273767 8608 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20250114 09:09:52.376340 8608 raft_consensus.cc:491] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:52.376435 8608 raft_consensus.cc:3054] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 15 FOLLOWER]: Advancing to term 16
I20250114 09:09:52.377120 8608 raft_consensus.cc:513] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17b287ff3546cebcd685f9965d063f" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 39703 } } peers { permanent_uuid: "7a637534fcda4d8d9c4521fcba5e77fd" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 34755 } } peers { permanent_uuid: "656e818763eb45d89ca10460f8d7a3b5" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 42225 } }
I20250114 09:09:52.377297 8608 leader_election.cc:290] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 16 election: Requested vote from peers 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703), 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755)
W20250114 09:09:52.377923 8566 leader_election.cc:336] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 7a637534fcda4d8d9c4521fcba5e77fd (127.7.109.131:34755): Network error: Client connection negotiation failed: client connection to 127.7.109.131:34755: connect: Connection refused (error 111)
W20250114 09:09:52.378000 8567 leader_election.cc:336] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 1d17b287ff3546cebcd685f9965d063f (127.7.109.130:39703): Network error: Client connection negotiation failed: client connection to 127.7.109.130:39703: connect: Connection refused (error 111)
I20250114 09:09:52.378044 8567 leader_election.cc:304] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 656e818763eb45d89ca10460f8d7a3b5; no voters: 1d17b287ff3546cebcd685f9965d063f, 7a637534fcda4d8d9c4521fcba5e77fd
I20250114 09:09:52.378124 8608 raft_consensus.cc:2743] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20250114 09:09:52.437050 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:52.437229 7606 tablet_replica.cc:331] stopping tablet replica
I20250114 09:09:52.437325 7606 raft_consensus.cc:2238] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 16 FOLLOWER]: Raft consensus shutting down.
I20250114 09:09:52.437384 7606 raft_consensus.cc:2267] T b19f15b21e194615a83631957faebe9a P 656e818763eb45d89ca10460f8d7a3b5 [term 16 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:52.449054 7606 tablet_server.cc:195] TabletServer@127.7.109.129:42225 shutdown complete.
[ OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (3242 ms)
[ RUN ] TimestampAdvancementITest.Kudu2463Test
I20250114 09:09:52.453295 7606 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.109.190:43171
I20250114 09:09:52.453433 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:52.455309 8639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:52.456048 7606 server_base.cc:1034] running on GCE node
W20250114 09:09:52.457005 8638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:52.457613 8641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:52.457774 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:52.457823 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:52.457839 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845792457838 us; error 0 us; skew 500 ppm
I20250114 09:09:52.458482 7606 webserver.cc:458] Webserver started at http://127.7.109.190:38643/ using document root <none> and password file <none>
I20250114 09:09:52.458580 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:52.458627 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:52.458678 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:52.459012 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/master-0-root/instance:
uuid: "b65a40b7fc3e4c189f38bcb8abe76375"
format_stamp: "Formatted at 2025-01-14 09:09:52 on dist-test-slave-msv5"
I20250114 09:09:52.459889 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:52.460269 8646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.460382 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.460423 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/master-0-root
uuid: "b65a40b7fc3e4c189f38bcb8abe76375"
format_stamp: "Formatted at 2025-01-14 09:09:52 on dist-test-slave-msv5"
I20250114 09:09:52.460459 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:52.473599 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:52.473799 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:52.476199 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.190:43171
I20250114 09:09:52.476747 8698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.190:43171 every 8 connection(s)
I20250114 09:09:52.476887 8699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:52.477838 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375: Bootstrap starting.
I20250114 09:09:52.478152 8699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:52.478662 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375: No bootstrap required, opened a new log
I20250114 09:09:52.478814 8699 raft_consensus.cc:357] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a40b7fc3e4c189f38bcb8abe76375" member_type: VOTER }
I20250114 09:09:52.478883 8699 raft_consensus.cc:383] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:52.478899 8699 raft_consensus.cc:738] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65a40b7fc3e4c189f38bcb8abe76375, State: Initialized, Role: FOLLOWER
I20250114 09:09:52.478957 8699 consensus_queue.cc:260] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a40b7fc3e4c189f38bcb8abe76375" member_type: VOTER }
I20250114 09:09:52.478997 8699 raft_consensus.cc:397] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 09:09:52.479027 8699 raft_consensus.cc:491] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 09:09:52.479056 8699 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:52.479563 8699 raft_consensus.cc:513] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a40b7fc3e4c189f38bcb8abe76375" member_type: VOTER }
I20250114 09:09:52.479624 8699 leader_election.cc:304] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b65a40b7fc3e4c189f38bcb8abe76375; no voters:
I20250114 09:09:52.479737 8699 leader_election.cc:290] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 09:09:52.479790 8702 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 09:09:52.479920 8699 sys_catalog.cc:564] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 09:09:52.479941 8702 raft_consensus.cc:695] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 1 LEADER]: Becoming Leader. State: Replica: b65a40b7fc3e4c189f38bcb8abe76375, State: Running, Role: LEADER
I20250114 09:09:52.479988 8702 consensus_queue.cc:237] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a40b7fc3e4c189f38bcb8abe76375" member_type: VOTER }
I20250114 09:09:52.480180 8704 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b65a40b7fc3e4c189f38bcb8abe76375. Latest consensus state: current_term: 1 leader_uuid: "b65a40b7fc3e4c189f38bcb8abe76375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a40b7fc3e4c189f38bcb8abe76375" member_type: VOTER } }
I20250114 09:09:52.480180 8703 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b65a40b7fc3e4c189f38bcb8abe76375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a40b7fc3e4c189f38bcb8abe76375" member_type: VOTER } }
I20250114 09:09:52.480261 8704 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:52.480269 8703 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:52.480428 8706 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 09:09:52.480592 8706 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 09:09:52.481103 8706 catalog_manager.cc:1348] Generated new cluster ID: 227e1edb7d3a4ec4ab7a279b971d7a2c
I20250114 09:09:52.481161 8706 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 09:09:52.481355 7606 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 09:09:52.485740 8706 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 09:09:52.486174 8706 catalog_manager.cc:1505] Loading token signing keys...
I20250114 09:09:52.492890 8706 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375: Generated new TSK 0
I20250114 09:09:52.492981 8706 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 09:09:52.496990 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:52.498863 8720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:52.499603 8723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:52.500171 8721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:52.500741 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:52.500823 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:52.500854 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:52.500865 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845792500866 us; error 0 us; skew 500 ppm
I20250114 09:09:52.501502 7606 webserver.cc:458] Webserver started at http://127.7.109.129:42297/ using document root <none> and password file <none>
I20250114 09:09:52.501576 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:52.501621 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:52.501668 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:52.501969 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root/instance:
uuid: "266fb9bb8b194462a4f2a232b62a6fbb"
format_stamp: "Formatted at 2025-01-14 09:09:52 on dist-test-slave-msv5"
I20250114 09:09:52.502779 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250114 09:09:52.503188 8728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.503305 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.503350 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root
uuid: "266fb9bb8b194462a4f2a232b62a6fbb"
format_stamp: "Formatted at 2025-01-14 09:09:52 on dist-test-slave-msv5"
I20250114 09:09:52.503386 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:52.523142 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:52.523370 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:52.523510 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:52.523701 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:52.523727 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.523756 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:52.523778 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.526418 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.129:45211
I20250114 09:09:52.526476 8791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.129:45211 every 8 connection(s)
I20250114 09:09:52.526749 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:52.528201 8796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:52.528718 8797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:52.529296 8799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:52.530263 8792 heartbeater.cc:346] Connected to a master server at 127.7.109.190:43171
I20250114 09:09:52.530284 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:52.530335 8792 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:52.530443 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250114 09:09:52.530449 8792 heartbeater.cc:510] Master 127.7.109.190:43171 requested a full tablet report, sending...
W20250114 09:09:52.530479 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:52.530498 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845792530498 us; error 0 us; skew 500 ppm
I20250114 09:09:52.530695 8663 ts_manager.cc:194] Registered new tserver with Master: 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129:45211)
I20250114 09:09:52.531139 7606 webserver.cc:458] Webserver started at http://127.7.109.130:38799/ using document root <none> and password file <none>
I20250114 09:09:52.531219 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:52.531263 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:52.531314 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:52.531448 8663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37630
I20250114 09:09:52.531615 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-1-root/instance:
uuid: "0536733d009e44cbb4e91def353316c9"
format_stamp: "Formatted at 2025-01-14 09:09:52 on dist-test-slave-msv5"
I20250114 09:09:52.532516 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250114 09:09:52.532975 8804 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.533092 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.533135 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-1-root
uuid: "0536733d009e44cbb4e91def353316c9"
format_stamp: "Formatted at 2025-01-14 09:09:52 on dist-test-slave-msv5"
I20250114 09:09:52.533179 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:52.549381 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:52.549620 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:52.549770 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:52.549955 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:52.549983 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.550010 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:52.550034 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.553035 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.130:40183
I20250114 09:09:52.553100 8867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.130:40183 every 8 connection(s)
I20250114 09:09:52.553417 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:52.555090 8871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:52.555671 8872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:52.556387 8868 heartbeater.cc:346] Connected to a master server at 127.7.109.190:43171
I20250114 09:09:52.556444 8868 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:52.556563 8868 heartbeater.cc:510] Master 127.7.109.190:43171 requested a full tablet report, sending...
W20250114 09:09:52.556725 8874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:52.556823 8663 ts_manager.cc:194] Registered new tserver with Master: 0536733d009e44cbb4e91def353316c9 (127.7.109.130:40183)
I20250114 09:09:52.557164 8663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37642
I20250114 09:09:52.557345 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:52.557469 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:52.557503 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:52.557524 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845792557524 us; error 0 us; skew 500 ppm
I20250114 09:09:52.558200 7606 webserver.cc:458] Webserver started at http://127.7.109.131:43695/ using document root <none> and password file <none>
I20250114 09:09:52.558269 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:52.558307 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:52.558341 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:52.558606 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-2-root/instance:
uuid: "3582c5383f2f436f954e0822ff0a273f"
format_stamp: "Formatted at 2025-01-14 09:09:52 on dist-test-slave-msv5"
I20250114 09:09:52.559373 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250114 09:09:52.559741 8879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.559844 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.559885 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-2-root
uuid: "3582c5383f2f436f954e0822ff0a273f"
format_stamp: "Formatted at 2025-01-14 09:09:52 on dist-test-slave-msv5"
I20250114 09:09:52.559921 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:52.570029 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:52.570251 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:52.570379 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:52.570557 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:52.570583 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.570605 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:52.570628 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:52.573673 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.131:41507
I20250114 09:09:52.573732 8942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.131:41507 every 8 connection(s)
I20250114 09:09:52.576695 8943 heartbeater.cc:346] Connected to a master server at 127.7.109.190:43171
I20250114 09:09:52.576758 8943 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:52.576838 8943 heartbeater.cc:510] Master 127.7.109.190:43171 requested a full tablet report, sending...
I20250114 09:09:52.577064 8663 ts_manager.cc:194] Registered new tserver with Master: 3582c5383f2f436f954e0822ff0a273f (127.7.109.131:41507)
I20250114 09:09:52.577126 7606 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003305571s
I20250114 09:09:52.577422 8663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37646
I20250114 09:09:52.583767 7606 test_util.cc:274] Using random seed: -327212972
I20250114 09:09:52.587736 8663 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37672:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 09:09:52.587880 8663 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 09:09:52.592339 8756 tablet_service.cc:1467] Processing CreateTablet for tablet d87c08b50ad146b7bff4cb4032767508 (DEFAULT_TABLE table=test-workload [id=72328f7416664242abac34edef592ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:52.592475 8756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87c08b50ad146b7bff4cb4032767508. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:52.593834 8832 tablet_service.cc:1467] Processing CreateTablet for tablet d87c08b50ad146b7bff4cb4032767508 (DEFAULT_TABLE table=test-workload [id=72328f7416664242abac34edef592ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:52.593932 8907 tablet_service.cc:1467] Processing CreateTablet for tablet d87c08b50ad146b7bff4cb4032767508 (DEFAULT_TABLE table=test-workload [id=72328f7416664242abac34edef592ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:52.593851 8964 tablet_bootstrap.cc:492] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Bootstrap starting.
I20250114 09:09:52.593976 8832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87c08b50ad146b7bff4cb4032767508. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:52.594064 8907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87c08b50ad146b7bff4cb4032767508. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:52.594370 8964 tablet_bootstrap.cc:654] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:52.595275 8967 tablet_bootstrap.cc:492] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9: Bootstrap starting.
I20250114 09:09:52.595583 8964 tablet_bootstrap.cc:492] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: No bootstrap required, opened a new log
I20250114 09:09:52.595628 8964 ts_tablet_manager.cc:1397] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250114 09:09:52.595641 8967 tablet_bootstrap.cc:654] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:52.595780 8964 raft_consensus.cc:357] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.595835 8964 raft_consensus.cc:383] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:52.595854 8964 raft_consensus.cc:738] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266fb9bb8b194462a4f2a232b62a6fbb, State: Initialized, Role: FOLLOWER
I20250114 09:09:52.595932 8964 consensus_queue.cc:260] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.596114 8964 ts_tablet_manager.cc:1428] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250114 09:09:52.596130 8792 heartbeater.cc:502] Master 127.7.109.190:43171 was elected leader, sending a full tablet report...
I20250114 09:09:52.595928 8966 tablet_bootstrap.cc:492] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f: Bootstrap starting.
I20250114 09:09:52.596181 8967 tablet_bootstrap.cc:492] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9: No bootstrap required, opened a new log
I20250114 09:09:52.596220 8967 ts_tablet_manager.cc:1397] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:52.596354 8967 raft_consensus.cc:357] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.596400 8967 raft_consensus.cc:383] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:52.596418 8967 raft_consensus.cc:738] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0536733d009e44cbb4e91def353316c9, State: Initialized, Role: FOLLOWER
I20250114 09:09:52.596462 8967 consensus_queue.cc:260] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.596630 8967 ts_tablet_manager.cc:1428] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250114 09:09:52.596657 8868 heartbeater.cc:502] Master 127.7.109.190:43171 was elected leader, sending a full tablet report...
I20250114 09:09:52.596652 8966 tablet_bootstrap.cc:654] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:52.597396 8966 tablet_bootstrap.cc:492] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f: No bootstrap required, opened a new log
I20250114 09:09:52.597441 8966 ts_tablet_manager.cc:1397] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250114 09:09:52.597566 8966 raft_consensus.cc:357] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.597610 8966 raft_consensus.cc:383] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:52.597628 8966 raft_consensus.cc:738] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3582c5383f2f436f954e0822ff0a273f, State: Initialized, Role: FOLLOWER
I20250114 09:09:52.597672 8966 consensus_queue.cc:260] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.597808 8966 ts_tablet_manager.cc:1428] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250114 09:09:52.597841 8943 heartbeater.cc:502] Master 127.7.109.190:43171 was elected leader, sending a full tablet report...
I20250114 09:09:52.668787 8970 raft_consensus.cc:491] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 09:09:52.668897 8970 raft_consensus.cc:513] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.669121 8970 leader_election.cc:290] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129:45211), 3582c5383f2f436f954e0822ff0a273f (127.7.109.131:41507)
I20250114 09:09:52.671761 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87c08b50ad146b7bff4cb4032767508" candidate_uuid: "0536733d009e44cbb4e91def353316c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3582c5383f2f436f954e0822ff0a273f" is_pre_election: true
I20250114 09:09:52.671861 8917 raft_consensus.cc:2463] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0536733d009e44cbb4e91def353316c9 in term 0.
I20250114 09:09:52.671900 8766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87c08b50ad146b7bff4cb4032767508" candidate_uuid: "0536733d009e44cbb4e91def353316c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" is_pre_election: true
I20250114 09:09:52.671993 8766 raft_consensus.cc:2463] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0536733d009e44cbb4e91def353316c9 in term 0.
I20250114 09:09:52.672019 8805 leader_election.cc:304] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0536733d009e44cbb4e91def353316c9, 3582c5383f2f436f954e0822ff0a273f; no voters:
I20250114 09:09:52.672099 8970 raft_consensus.cc:2798] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 09:09:52.672147 8970 raft_consensus.cc:491] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 09:09:52.672164 8970 raft_consensus.cc:3054] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:52.672806 8970 raft_consensus.cc:513] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.672931 8970 leader_election.cc:290] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [CANDIDATE]: Term 1 election: Requested vote from peers 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129:45211), 3582c5383f2f436f954e0822ff0a273f (127.7.109.131:41507)
I20250114 09:09:52.673040 8766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87c08b50ad146b7bff4cb4032767508" candidate_uuid: "0536733d009e44cbb4e91def353316c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266fb9bb8b194462a4f2a232b62a6fbb"
I20250114 09:09:52.673089 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87c08b50ad146b7bff4cb4032767508" candidate_uuid: "0536733d009e44cbb4e91def353316c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3582c5383f2f436f954e0822ff0a273f"
I20250114 09:09:52.673094 8766 raft_consensus.cc:3054] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:52.673173 8917 raft_consensus.cc:3054] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:52.673759 8766 raft_consensus.cc:2463] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0536733d009e44cbb4e91def353316c9 in term 1.
I20250114 09:09:52.673771 8917 raft_consensus.cc:2463] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0536733d009e44cbb4e91def353316c9 in term 1.
I20250114 09:09:52.673929 8805 leader_election.cc:304] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0536733d009e44cbb4e91def353316c9, 3582c5383f2f436f954e0822ff0a273f; no voters:
I20250114 09:09:52.674022 8970 raft_consensus.cc:2798] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 09:09:52.674175 8970 raft_consensus.cc:695] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Becoming Leader. State: Replica: 0536733d009e44cbb4e91def353316c9, State: Running, Role: LEADER
I20250114 09:09:52.674226 8970 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.674748 8663 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0536733d009e44cbb4e91def353316c9 (127.7.109.130). New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: UNKNOWN } } }
I20250114 09:09:52.683990 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 09:09:52.684006 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 09:09:52.684219 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 09:09:52.684343 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 09:09:52.702258 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.702667 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250114 09:09:52.702680 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250114 09:09:52.702875 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250114 09:09:52.702970 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250114 09:09:52.703552 8982 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.703655 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.704721 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.704726 8662 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index -1 to 17, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.704970 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.705404 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
W20250114 09:09:52.705530 8660 catalog_manager.cc:5204] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 17: no extra replica candidate found for tablet d87c08b50ad146b7bff4cb4032767508 (table test-workload [id=72328f7416664242abac34edef592ded]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250114 09:09:52.705685 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250114 09:09:52.705907 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250114 09:09:52.706040 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250114 09:09:52.706486 8982 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.706612 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.706638 8765 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.707520 8662 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index 17 to 18, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.707747 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.708035 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250114 09:09:52.708174 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 09:09:52.708196 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250114 09:09:52.708369 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 09:09:52.708700 8974 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.708815 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.709556 8663 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index 18 to 19, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.709709 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.709816 8765 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.710129 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250114 09:09:52.710301 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 09:09:52.710723 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250114 09:09:52.710888 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 09:09:52.710986 8974 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.711143 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.711704 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.711839 8660 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: config changed from index 19 to 20, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:52.711923 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.712193 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250114 09:09:52.712344 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 09:09:52.712803 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250114 09:09:52.713145 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 09:09:52.713539 8982 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.713641 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.714334 8663 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index 20 to 21, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.714377 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.714746 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.715143 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250114 09:09:52.715277 8970 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250114 09:09:52.715365 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250114 09:09:52.715503 8970 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250114 09:09:52.715871 8982 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.716001 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.716051 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.716835 8660 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index 21 to 22, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.716922 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.717221 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250114 09:09:52.717199 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250114 09:09:52.717499 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250114 09:09:52.717566 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250114 09:09:52.718067 8974 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.718178 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.718194 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.718940 8662 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index 22 to 23, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.719228 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.719568 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250114 09:09:52.719705 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250114 09:09:52.719730 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250114 09:09:52.719871 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250114 09:09:52.720118 8970 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.720937 8660 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: config changed from index 23 to 24, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:52.720947 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.720947 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.721263 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.721782 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250114 09:09:52.722106 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250114 09:09:52.722182 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250114 09:09:52.722415 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250114 09:09:52.722649 8974 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.723415 8660 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: config changed from index 24 to 25, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:52.723567 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.723646 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.723989 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250114 09:09:52.724059 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.724110 8970 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20250114 09:09:52.724844 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250114 09:09:52.725031 8970 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250114 09:09:52.725376 8982 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.725492 8916 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.725620 8765 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.726450 8663 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb reported cstate change: config changed from index 25 to 26, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.727101 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.727412 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250114 09:09:52.727464 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250114 09:09:52.727587 8999 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250114 09:09:52.727665 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250114 09:09:52.728103 8974 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.728217 8916 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.729097 8660 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index 26 to 27, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.729112 8765 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.729338 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.729727 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250114 09:09:52.730197 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250114 09:09:52.730264 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250114 09:09:52.730441 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250114 09:09:52.730659 8974 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.730751 8916 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.731472 8663 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index 27 to 28, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.731537 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.731644 8765 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.731876 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250114 09:09:52.732020 8982 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250114 09:09:52.732453 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250114 09:09:52.732538 8999 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.732669 8916 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.732715 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250114 09:09:52.733037 8765 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.733428 8660 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: config changed from index 28 to 29, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:52.733656 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.733958 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250114 09:09:52.734090 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250114 09:09:52.734088 8970 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250114 09:09:52.734287 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250114 09:09:52.734668 8970 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.734776 8916 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.735486 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.735509 8662 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index 29 to 30, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.735617 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.736096 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250114 09:09:52.736241 8999 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250114 09:09:52.736407 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250114 09:09:52.736578 8999 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250114 09:09:52.736994 8982 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.737099 8916 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.737290 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.737953 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.737934 8663 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: config changed from index 30 to 31, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:52.738291 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250114 09:09:52.738454 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250114 09:09:52.738462 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250114 09:09:52.738775 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250114 09:09:52.738950 8970 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.739465 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.739691 8916 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.739791 8663 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: config changed from index 31 to 32, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:52.739830 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.740522 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250114 09:09:52.740736 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250114 09:09:52.740761 8999 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250114 09:09:52.740885 8999 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250114 09:09:52.741329 8982 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.741456 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.742312 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.742357 8765 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.742314 8660 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: config changed from index 32 to 33, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:52.742698 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20250114 09:09:52.742870 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250114 09:09:52.743352 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20250114 09:09:52.743480 8970 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.743559 8974 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250114 09:09:52.743619 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.744266 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.744297 8660 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: config changed from index 33 to 34, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:52.744385 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.744961 8916 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20250114 09:09:52.745090 8650 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 21: aborting the task: latest config opid_index 34; task opid_index 21
I20250114 09:09:52.745266 8999 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250114 09:09:52.745308 8765 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20250114 09:09:52.745555 8999 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250114 09:09:52.745707 8970 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.745903 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.746145 8765 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.746559 8842 consensus_queue.cc:237] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:52.746537 8660 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 reported cstate change: config changed from index 34 to 35, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: NON_VOTER last_known_addr { host: "127.7.109.129" port: 45211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:52.746977 8917 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20250114 09:09:52.747162 8999 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250114 09:09:52.747285 8766 raft_consensus.cc:1270] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 LEARNER]: Refusing update from remote peer 0536733d009e44cbb4e91def353316c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20250114 09:09:52.747421 8999 consensus_queue.cc:1035] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250114 09:09:52.747643 8999 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.747737 8917 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.748421 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:52.748533 8663 catalog_manager.cc:5526] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f reported cstate change: config changed from index 35 to 36, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0536733d009e44cbb4e91def353316c9" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.748817 8766 raft_consensus.cc:2949] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 266fb9bb8b194462a4f2a232b62a6fbb (127.7.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } } }
I20250114 09:09:52.749620 7606 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 09:09:52.749683 7606 log_reader.cc:385] T d87c08b50ad146b7bff4cb4032767508: removed 1 log segments from log reader
I20250114 09:09:52.749707 7606 log.cc:1078] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Deleting log segment in path: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root/wals/d87c08b50ad146b7bff4cb4032767508/wal-000000001 (ops 1-16)
I20250114 09:09:52.750276 7606 timestamp_advancement-itest.cc:268] Shutting down cluster...
I20250114 09:09:52.750327 7606 master.cc:537] Master@127.7.109.190:43171 shutting down...
I20250114 09:09:52.750552 8647 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 23: aborting the task: latest config opid_index 36; task opid_index 23
I20250114 09:09:52.757737 8647 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 19: aborting the task: latest config opid_index 36; task opid_index 19
I20250114 09:09:52.758911 8647 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 29: aborting the task: latest config opid_index 36; task opid_index 29
I20250114 09:09:52.761045 8647 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 27: aborting the task: latest config opid_index 36; task opid_index 27
I20250114 09:09:52.762943 8648 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 17: aborting the task: latest config opid_index 36; task opid_index 17
I20250114 09:09:52.770200 8647 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 35: aborting the task: latest config opid_index 36; task opid_index 35
I20250114 09:09:52.787384 8647 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 25: aborting the task: latest config opid_index 36; task opid_index 25
I20250114 09:09:52.791558 8647 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 31: aborting the task: latest config opid_index 36; task opid_index 31
I20250114 09:09:52.793702 8647 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d87c08b50ad146b7bff4cb4032767508 with cas_config_opid_index 33: aborting the task: latest config opid_index 36; task opid_index 33
I20250114 09:09:52.974699 7606 raft_consensus.cc:2238] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 1 LEADER]: Raft consensus shutting down.
I20250114 09:09:52.974813 7606 raft_consensus.cc:2267] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:52.974917 7606 tablet_replica.cc:331] T 00000000000000000000000000000000 P b65a40b7fc3e4c189f38bcb8abe76375: stopping tablet replica
I20250114 09:09:52.987192 7606 master.cc:559] Master@127.7.109.190:43171 shutdown complete.
I20250114 09:09:52.988387 7606 tablet_server.cc:178] TabletServer@127.7.109.129:0 shutting down...
I20250114 09:09:53.027184 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:53.027354 7606 tablet_replica.cc:331] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: stopping tablet replica
I20250114 09:09:53.027422 7606 raft_consensus.cc:2238] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 09:09:53.027482 7606 raft_consensus.cc:2267] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:53.039425 7606 tablet_server.cc:195] TabletServer@127.7.109.129:0 shutdown complete.
I20250114 09:09:53.040446 7606 tablet_server.cc:178] TabletServer@127.7.109.130:0 shutting down...
I20250114 09:09:53.053741 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:53.053911 7606 tablet_replica.cc:331] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9: stopping tablet replica
I20250114 09:09:53.053970 7606 raft_consensus.cc:2238] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 LEADER]: Raft consensus shutting down.
I20250114 09:09:53.054052 7606 raft_consensus.cc:2267] T d87c08b50ad146b7bff4cb4032767508 P 0536733d009e44cbb4e91def353316c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:53.066367 7606 tablet_server.cc:195] TabletServer@127.7.109.130:0 shutdown complete.
I20250114 09:09:53.067240 7606 tablet_server.cc:178] TabletServer@127.7.109.131:0 shutting down...
I20250114 09:09:53.074340 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:53.074505 7606 tablet_replica.cc:331] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f: stopping tablet replica
I20250114 09:09:53.074565 7606 raft_consensus.cc:2238] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 09:09:53.074698 7606 raft_consensus.cc:2267] T d87c08b50ad146b7bff4cb4032767508 P 3582c5383f2f436f954e0822ff0a273f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:53.086598 7606 tablet_server.cc:195] TabletServer@127.7.109.131:0 shutdown complete.
I20250114 09:09:53.087484 7606 timestamp_advancement-itest.cc:276] Restarting single tablet server...
I20250114 09:09:53.087603 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:53.089601 9004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.090229 9002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.090981 9001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:53.091606 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:53.091709 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:53.091737 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:53.091749 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845793091749 us; error 0 us; skew 500 ppm
I20250114 09:09:53.092386 7606 webserver.cc:458] Webserver started at http://127.7.109.129:42297/ using document root <none> and password file <none>
I20250114 09:09:53.092458 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:53.092479 7606 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 09:09:53.093133 7606 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.094297 9012 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.094404 7606 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:53.094456 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root
uuid: "266fb9bb8b194462a4f2a232b62a6fbb"
format_stamp: "Formatted at 2025-01-14 09:09:52 on dist-test-slave-msv5"
I20250114 09:09:53.094506 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736845785692342-7606-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5293
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:53.098160 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:53.098320 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:53.098449 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:53.099201 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 09:09:53.099231 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:53.099619 7606 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 09:09:53.099645 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.099663 9019 tablet_bootstrap.cc:492] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Bootstrap starting.
I20250114 09:09:53.101378 9019 tablet_bootstrap.cc:492] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 09:09:53.101591 9019 tablet_bootstrap.cc:492] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Bootstrap complete.
I20250114 09:09:53.101681 9019 ts_tablet_manager.cc:1397] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250114 09:09:53.101801 9019 raft_consensus.cc:357] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:53.101871 9019 raft_consensus.cc:738] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266fb9bb8b194462a4f2a232b62a6fbb, State: Initialized, Role: FOLLOWER
I20250114 09:09:53.101918 9019 consensus_queue.cc:260] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "266fb9bb8b194462a4f2a232b62a6fbb" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 45211 } } peers { permanent_uuid: "0536733d009e44cbb4e91def353316c9" member_type: VOTER last_known_addr { host: "127.7.109.130" port: 40183 } } peers { permanent_uuid: "3582c5383f2f436f954e0822ff0a273f" member_type: VOTER last_known_addr { host: "127.7.109.131" port: 41507 } }
I20250114 09:09:53.102074 9019 ts_tablet_manager.cc:1428] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s
I20250114 09:09:53.102627 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.129:45211
I20250114 09:09:53.104974 9078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.129:45211 every 8 connection(s)
W20250114 09:09:53.105666 9079 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250114 09:09:53.110109 9032 tablet_service.cc:2908] Rejecting scan request for tablet d87c08b50ad146b7bff4cb4032767508: Uninitialized: clean time has not yet been initialized
I20250114 09:09:53.110347 7606 timestamp_advancement-itest.cc:441] Got response: error { code: TABLET_NOT_RUNNING status { code: UNINITIALIZED message: "clean time has not yet been initialized" } }
I20250114 09:09:53.110674 7606 tablet_server.cc:178] TabletServer@127.7.109.129:45211 shutting down...
I20250114 09:09:53.355393 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:53.355569 7606 tablet_replica.cc:331] stopping tablet replica
I20250114 09:09:53.355640 7606 raft_consensus.cc:2238] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 09:09:53.355698 7606 raft_consensus.cc:2267] T d87c08b50ad146b7bff4cb4032767508 P 266fb9bb8b194462a4f2a232b62a6fbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:53.367288 7606 tablet_server.cc:195] TabletServer@127.7.109.129:45211 shutdown complete.
[ OK ] TimestampAdvancementITest.Kudu2463Test (918 ms)
[ RUN ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp
I20250114 09:09:53.371383 7606 timestamp_advancement-itest.cc:454] Running with 1 replicas
I20250114 09:09:53.371533 7606 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.109.190:33679
I20250114 09:09:53.371673 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:53.373653 9083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.374222 9086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.374709 9084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:53.375717 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:53.375808 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:53.375833 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:53.375850 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845793375850 us; error 0 us; skew 500 ppm
I20250114 09:09:53.376468 7606 webserver.cc:458] Webserver started at http://127.7.109.190:38215/ using document root <none> and password file <none>
I20250114 09:09:53.376577 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:53.376621 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:53.376667 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:53.376948 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/master-0-root/instance:
uuid: "2e2df6bdb2344e99881444b88fe78cf2"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.377737 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250114 09:09:53.378183 9091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.378305 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.378348 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/master-0-root
uuid: "2e2df6bdb2344e99881444b88fe78cf2"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.378396 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:53.388486 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:53.388731 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:53.392678 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.190:33679
I20250114 09:09:53.393199 9143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.190:33679 every 8 connection(s)
I20250114 09:09:53.393333 9144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:53.394356 9144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2: Bootstrap starting.
I20250114 09:09:53.394672 9144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:53.395165 9144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2: No bootstrap required, opened a new log
I20250114 09:09:53.395280 9144 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER }
I20250114 09:09:53.395334 9144 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:53.395346 9144 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2df6bdb2344e99881444b88fe78cf2, State: Initialized, Role: FOLLOWER
I20250114 09:09:53.395396 9144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER }
I20250114 09:09:53.395432 9144 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 09:09:53.395454 9144 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 09:09:53.395480 9144 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:53.395951 9144 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER }
I20250114 09:09:53.396003 9144 leader_election.cc:304] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e2df6bdb2344e99881444b88fe78cf2; no voters:
I20250114 09:09:53.396107 9144 leader_election.cc:290] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 09:09:53.396179 9147 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 09:09:53.396237 9144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 09:09:53.396323 9147 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 LEADER]: Becoming Leader. State: Replica: 2e2df6bdb2344e99881444b88fe78cf2, State: Running, Role: LEADER
I20250114 09:09:53.396378 9147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER }
I20250114 09:09:53.396624 9148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e2df6bdb2344e99881444b88fe78cf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER } }
I20250114 09:09:53.396638 9149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e2df6bdb2344e99881444b88fe78cf2. Latest consensus state: current_term: 1 leader_uuid: "2e2df6bdb2344e99881444b88fe78cf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER } }
I20250114 09:09:53.396764 9149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:53.396939 9148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:53.396937 9155 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 09:09:53.397153 9155 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 09:09:53.397504 7606 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 09:09:53.397719 9155 catalog_manager.cc:1348] Generated new cluster ID: d5bfaa61009a4a2ab804670c20991617
I20250114 09:09:53.397760 9155 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 09:09:53.405092 9155 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 09:09:53.405519 9155 catalog_manager.cc:1505] Loading token signing keys...
I20250114 09:09:53.413671 9155 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2: Generated new TSK 0
I20250114 09:09:53.413762 9155 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 09:09:53.429312 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:53.431298 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.431819 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.432327 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:53.433337 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:53.433468 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:53.433497 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:53.433514 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845793433514 us; error 0 us; skew 500 ppm
I20250114 09:09:53.434202 7606 webserver.cc:458] Webserver started at http://127.7.109.129:34077/ using document root <none> and password file <none>
I20250114 09:09:53.434285 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:53.434334 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:53.434382 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:53.434656 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-0-root/instance:
uuid: "c1cc7fb48c2c42cca829f60797f1826c"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.435464 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:53.435854 9173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.435984 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.436025 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-0-root
uuid: "c1cc7fb48c2c42cca829f60797f1826c"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.436070 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:53.458086 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:53.458319 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:53.458464 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:53.458673 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:53.458711 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.458740 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:53.458762 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.461405 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.129:46481
I20250114 09:09:53.461442 9236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.129:46481 every 8 connection(s)
I20250114 09:09:53.464609 9237 heartbeater.cc:346] Connected to a master server at 127.7.109.190:33679
I20250114 09:09:53.464668 9237 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:53.464751 9237 heartbeater.cc:510] Master 127.7.109.190:33679 requested a full tablet report, sending...
I20250114 09:09:53.465020 9108 ts_manager.cc:194] Registered new tserver with Master: c1cc7fb48c2c42cca829f60797f1826c (127.7.109.129:46481)
I20250114 09:09:53.465548 9108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57792
I20250114 09:09:53.465945 7606 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004349635s
I20250114 09:09:53.472384 7606 test_util.cc:274] Using random seed: -326324353
I20250114 09:09:53.476426 9108 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 09:09:53.480165 9201 tablet_service.cc:1467] Processing CreateTablet for tablet 8370070e2e744402bb0935b777caf79e (DEFAULT_TABLE table=test-workload [id=f963019ab2cc42cca25ef727d4d852a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 09:09:53.480301 9201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8370070e2e744402bb0935b777caf79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 09:09:53.481536 9255 tablet_bootstrap.cc:492] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: Bootstrap starting.
I20250114 09:09:53.481853 9255 tablet_bootstrap.cc:654] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: Neither blocks nor log segments found. Creating new log.
I20250114 09:09:53.482303 9255 tablet_bootstrap.cc:492] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: No bootstrap required, opened a new log
I20250114 09:09:53.482342 9255 ts_tablet_manager.cc:1397] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:53.482443 9255 raft_consensus.cc:357] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } }
I20250114 09:09:53.482492 9255 raft_consensus.cc:383] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 09:09:53.482504 9255 raft_consensus.cc:738] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cc7fb48c2c42cca829f60797f1826c, State: Initialized, Role: FOLLOWER
I20250114 09:09:53.482551 9255 consensus_queue.cc:260] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } }
I20250114 09:09:53.482589 9255 raft_consensus.cc:397] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 09:09:53.482604 9255 raft_consensus.cc:491] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 09:09:53.482630 9255 raft_consensus.cc:3054] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 0 FOLLOWER]: Advancing to term 1
I20250114 09:09:53.483136 9255 raft_consensus.cc:513] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } }
I20250114 09:09:53.483207 9255 leader_election.cc:304] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1cc7fb48c2c42cca829f60797f1826c; no voters:
I20250114 09:09:53.483326 9255 leader_election.cc:290] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 09:09:53.483381 9257 raft_consensus.cc:2798] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 FOLLOWER]: Leader election won for term 1
I20250114 09:09:53.483464 9255 ts_tablet_manager.cc:1428] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:53.483507 9257 raft_consensus.cc:695] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 LEADER]: Becoming Leader. State: Replica: c1cc7fb48c2c42cca829f60797f1826c, State: Running, Role: LEADER
I20250114 09:09:53.483534 9237 heartbeater.cc:502] Master 127.7.109.190:33679 was elected leader, sending a full tablet report...
I20250114 09:09:53.483562 9257 consensus_queue.cc:237] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } }
I20250114 09:09:53.484001 9108 catalog_manager.cc:5526] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c reported cstate change: term changed from 0 to 1, leader changed from <none> to c1cc7fb48c2c42cca829f60797f1826c (127.7.109.129). New cstate: current_term: 1 leader_uuid: "c1cc7fb48c2c42cca829f60797f1826c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:53.486436 7606 tablet_server.cc:178] TabletServer@127.7.109.129:0 shutting down...
I20250114 09:09:53.488658 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:53.488824 7606 tablet_replica.cc:331] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: stopping tablet replica
I20250114 09:09:53.488881 7606 raft_consensus.cc:2238] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 LEADER]: Raft consensus shutting down.
I20250114 09:09:53.488934 7606 raft_consensus.cc:2267] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:53.500550 7606 tablet_server.cc:195] TabletServer@127.7.109.129:0 shutdown complete.
I20250114 09:09:53.501499 7606 master.cc:537] Master@127.7.109.190:33679 shutting down...
I20250114 09:09:53.503472 7606 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 09:09:53.503545 7606 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:53.503629 7606 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2: stopping tablet replica
I20250114 09:09:53.515496 7606 master.cc:559] Master@127.7.109.190:33679 shutdown complete.
I20250114 09:09:53.517148 7606 timestamp_advancement-itest.cc:454] Running with 3 replicas
I20250114 09:09:53.517292 7606 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.109.190:44269
I20250114 09:09:53.517385 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:53.519071 9262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.519668 9265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:53.520682 7606 server_base.cc:1034] running on GCE node
W20250114 09:09:53.521385 9263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:53.521509 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:53.521539 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:53.521555 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845793521556 us; error 0 us; skew 500 ppm
I20250114 09:09:53.522166 7606 webserver.cc:458] Webserver started at http://127.7.109.190:40661/ using document root <none> and password file <none>
I20250114 09:09:53.522245 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:53.522276 7606 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 09:09:53.522778 7606 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.523181 9270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.523298 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.523339 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/master-0-root
uuid: "2e2df6bdb2344e99881444b88fe78cf2"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.523391 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:53.532684 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:53.532874 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:53.535356 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.190:44269
I20250114 09:09:53.536034 9322 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.190:44269 every 8 connection(s)
I20250114 09:09:53.536633 9323 sys_catalog.cc:263] Verifying existing consensus state
I20250114 09:09:53.536984 9323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2: Bootstrap starting.
I20250114 09:09:53.538323 9323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 09:09:53.538528 9323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2: Bootstrap complete.
I20250114 09:09:53.538734 9323 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER }
I20250114 09:09:53.538791 9323 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2df6bdb2344e99881444b88fe78cf2, State: Initialized, Role: FOLLOWER
I20250114 09:09:53.538831 9323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER }
I20250114 09:09:53.538869 9323 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 09:09:53.538901 9323 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 09:09:53.538921 9323 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 1 FOLLOWER]: Advancing to term 2
I20250114 09:09:53.539402 9323 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER }
I20250114 09:09:53.539454 9323 leader_election.cc:304] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e2df6bdb2344e99881444b88fe78cf2; no voters:
I20250114 09:09:53.539556 9323 leader_election.cc:290] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250114 09:09:53.539604 9326 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 09:09:53.539696 9323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 09:09:53.539742 9326 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 2 LEADER]: Becoming Leader. State: Replica: 2e2df6bdb2344e99881444b88fe78cf2, State: Running, Role: LEADER
I20250114 09:09:53.539793 9326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER }
I20250114 09:09:53.539990 9327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2e2df6bdb2344e99881444b88fe78cf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER } }
I20250114 09:09:53.539995 9328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e2df6bdb2344e99881444b88fe78cf2. Latest consensus state: current_term: 2 leader_uuid: "2e2df6bdb2344e99881444b88fe78cf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2df6bdb2344e99881444b88fe78cf2" member_type: VOTER } }
I20250114 09:09:53.540067 9327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:53.540083 9328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [sys.catalog]: This master's current role is: LEADER
I20250114 09:09:53.540266 9332 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 09:09:53.540486 9332 catalog_manager.cc:670] Loaded metadata for table test-workload [id=f963019ab2cc42cca25ef727d4d852a5]
I20250114 09:09:53.540642 9332 tablet_loader.cc:96] loaded metadata for tablet 8370070e2e744402bb0935b777caf79e (table test-workload [id=f963019ab2cc42cca25ef727d4d852a5])
I20250114 09:09:53.540680 9332 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 09:09:53.540745 9332 catalog_manager.cc:1260] Loaded cluster ID: d5bfaa61009a4a2ab804670c20991617
I20250114 09:09:53.540773 9332 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 09:09:53.540977 7606 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 09:09:53.541558 9332 catalog_manager.cc:1505] Loading token signing keys...
I20250114 09:09:53.541657 9332 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2: Loaded TSK: 0
I20250114 09:09:53.541735 9332 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 09:09:53.542246 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:53.543943 9343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.544646 9344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.545226 9346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:53.545811 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:53.545886 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:53.545917 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:53.545934 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845793545934 us; error 0 us; skew 500 ppm
I20250114 09:09:53.546505 7606 webserver.cc:458] Webserver started at http://127.7.109.129:33997/ using document root <none> and password file <none>
I20250114 09:09:53.546568 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:53.546602 7606 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 09:09:53.547082 7606 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.547482 9351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.547570 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.547608 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-0-root
uuid: "c1cc7fb48c2c42cca829f60797f1826c"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.547650 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:53.564888 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:53.565169 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:53.565321 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:53.565601 9358 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250114 09:09:53.566176 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 09:09:53.566213 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250114 09:09:53.566242 7606 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250114 09:09:53.566632 7606 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 09:09:53.566661 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.566669 9358 tablet_bootstrap.cc:492] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: Bootstrap starting.
I20250114 09:09:53.567701 9358 tablet_bootstrap.cc:492] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 09:09:53.567899 9358 tablet_bootstrap.cc:492] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: Bootstrap complete.
I20250114 09:09:53.568002 9358 ts_tablet_manager.cc:1397] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:53.568126 9358 raft_consensus.cc:357] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } }
I20250114 09:09:53.568185 9358 raft_consensus.cc:738] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cc7fb48c2c42cca829f60797f1826c, State: Initialized, Role: FOLLOWER
I20250114 09:09:53.568235 9358 consensus_queue.cc:260] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } }
I20250114 09:09:53.568269 9358 raft_consensus.cc:397] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 09:09:53.568289 9358 raft_consensus.cc:491] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 09:09:53.568315 9358 raft_consensus.cc:3054] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 1 FOLLOWER]: Advancing to term 2
I20250114 09:09:53.568919 9358 raft_consensus.cc:513] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } }
I20250114 09:09:53.568993 9358 leader_election.cc:304] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1cc7fb48c2c42cca829f60797f1826c; no voters:
I20250114 09:09:53.569099 9358 leader_election.cc:290] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [CANDIDATE]: Term 2 election: Requested vote from peers
I20250114 09:09:53.569159 9402 raft_consensus.cc:2798] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 2 FOLLOWER]: Leader election won for term 2
I20250114 09:09:53.569224 9358 ts_tablet_manager.cc:1428] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 09:09:53.569272 9402 raft_consensus.cc:695] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 2 LEADER]: Becoming Leader. State: Replica: c1cc7fb48c2c42cca829f60797f1826c, State: Running, Role: LEADER
I20250114 09:09:53.569331 9402 consensus_queue.cc:237] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } }
I20250114 09:09:53.570029 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.129:32909
I20250114 09:09:53.570076 9421 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.129:32909 every 8 connection(s)
I20250114 09:09:53.570372 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:53.571905 9426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.572448 9429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.573040 9427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:53.573572 9422 heartbeater.cc:346] Connected to a master server at 127.7.109.190:44269
I20250114 09:09:53.573617 9422 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:53.573690 9422 heartbeater.cc:510] Master 127.7.109.190:44269 requested a full tablet report, sending...
I20250114 09:09:53.573951 9287 ts_manager.cc:194] Registered new tserver with Master: c1cc7fb48c2c42cca829f60797f1826c (127.7.109.129:32909)
I20250114 09:09:53.574009 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:53.574026 9287 catalog_manager.cc:5526] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c1cc7fb48c2c42cca829f60797f1826c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cc7fb48c2c42cca829f60797f1826c" member_type: VOTER last_known_addr { host: "127.7.109.129" port: 46481 } health_report { overall_health: HEALTHY } } }
I20250114 09:09:53.574146 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 09:09:53.574169 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:53.574182 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845793574182 us; error 0 us; skew 500 ppm
I20250114 09:09:53.574831 7606 webserver.cc:458] Webserver started at http://127.7.109.130:35337/ using document root <none> and password file <none>
I20250114 09:09:53.574908 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:53.574941 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:53.574972 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:53.575203 9287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55140
I20250114 09:09:53.575259 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-1-root/instance:
uuid: "a0227d26b52b4e9cbaab46b55095663b"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.575661 9422 heartbeater.cc:502] Master 127.7.109.190:44269 was elected leader, sending a full tablet report...
I20250114 09:09:53.576222 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250114 09:09:53.576656 9435 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.576763 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.576807 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-1-root
uuid: "a0227d26b52b4e9cbaab46b55095663b"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.576846 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:53.589462 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:53.589699 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:53.589843 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:53.590035 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:53.590065 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.590086 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:53.590111 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.593163 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.130:35699
I20250114 09:09:53.593220 9498 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.130:35699 every 8 connection(s)
I20250114 09:09:53.593501 7606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 09:09:53.595167 9503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 09:09:53.595916 9502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:53.596452 9499 heartbeater.cc:346] Connected to a master server at 127.7.109.190:44269
W20250114 09:09:53.596539 9505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 09:09:53.596573 9499 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:53.596689 9499 heartbeater.cc:510] Master 127.7.109.190:44269 requested a full tablet report, sending...
I20250114 09:09:53.596930 9287 ts_manager.cc:194] Registered new tserver with Master: a0227d26b52b4e9cbaab46b55095663b (127.7.109.130:35699)
I20250114 09:09:53.597204 7606 server_base.cc:1034] running on GCE node
I20250114 09:09:53.597343 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250114 09:09:53.597366 9287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55148
W20250114 09:09:53.597373 7606 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 09:09:53.597429 7606 hybrid_clock.cc:648] HybridClock initialized: now 1736845793597429 us; error 0 us; skew 500 ppm
I20250114 09:09:53.598096 7606 webserver.cc:458] Webserver started at http://127.7.109.131:34655/ using document root <none> and password file <none>
I20250114 09:09:53.598172 7606 fs_manager.cc:362] Metadata directory not provided
I20250114 09:09:53.598209 7606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 09:09:53.598242 7606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 09:09:53.598469 7606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-2-root/instance:
uuid: "feb7dee20fac4de59e61433ddc676d2f"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.599224 7606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250114 09:09:53.599589 9510 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.599735 7606 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.599776 7606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-2-root
uuid: "feb7dee20fac4de59e61433ddc676d2f"
format_stamp: "Formatted at 2025-01-14 09:09:53 on dist-test-slave-msv5"
I20250114 09:09:53.599819 7606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskXyX3es/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736845785692342-7606-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 09:09:53.603461 7606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 09:09:53.603683 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 09:09:53.603835 7606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 09:09:53.604018 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 09:09:53.604044 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.604063 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 09:09:53.604086 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 09:09:53.607132 7606 rpc_server.cc:307] RPC server started. Bound to: 127.7.109.131:44615
I20250114 09:09:53.607182 9573 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.109.131:44615 every 8 connection(s)
I20250114 09:09:53.609932 9574 heartbeater.cc:346] Connected to a master server at 127.7.109.190:44269
I20250114 09:09:53.609989 9574 heartbeater.cc:463] Registering TS with master...
I20250114 09:09:53.610087 9574 heartbeater.cc:510] Master 127.7.109.190:44269 requested a full tablet report, sending...
I20250114 09:09:53.610304 9287 ts_manager.cc:194] Registered new tserver with Master: feb7dee20fac4de59e61433ddc676d2f (127.7.109.131:44615)
I20250114 09:09:53.610596 7606 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003313039s
I20250114 09:09:53.610649 9287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55150
I20250114 09:09:53.617169 7606 test_util.cc:274] Using random seed: -326179569
I20250114 09:09:53.621518 7606 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250114 09:09:53.621591 7606 tablet_server.cc:178] TabletServer@127.7.109.129:0 shutting down...
I20250114 09:09:53.623915 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:53.624056 7606 tablet_replica.cc:331] stopping tablet replica
I20250114 09:09:53.624099 7606 raft_consensus.cc:2238] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 2 LEADER]: Raft consensus shutting down.
I20250114 09:09:53.624143 7606 raft_consensus.cc:2267] T 8370070e2e744402bb0935b777caf79e P c1cc7fb48c2c42cca829f60797f1826c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:53.635793 7606 tablet_server.cc:195] TabletServer@127.7.109.129:0 shutdown complete.
I20250114 09:09:53.636729 7606 tablet_server.cc:178] TabletServer@127.7.109.130:0 shutting down...
I20250114 09:09:53.639014 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:53.650362 7606 tablet_server.cc:195] TabletServer@127.7.109.130:0 shutdown complete.
I20250114 09:09:53.651154 7606 tablet_server.cc:178] TabletServer@127.7.109.131:0 shutting down...
I20250114 09:09:53.653491 7606 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 09:09:53.664757 7606 tablet_server.cc:195] TabletServer@127.7.109.131:0 shutdown complete.
I20250114 09:09:53.665536 7606 master.cc:537] Master@127.7.109.190:44269 shutting down...
I20250114 09:09:53.667516 7606 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 2 LEADER]: Raft consensus shutting down.
I20250114 09:09:53.667593 7606 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2e2df6bdb2344e99881444b88fe78cf2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 09:09:53.667658 7606 tablet_replica.cc:331] stopping tablet replica
I20250114 09:09:53.679550 7606 master.cc:559] Master@127.7.109.190:44269 shutdown complete.
[ OK ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp (312 ms)
[----------] 4 tests from TimestampAdvancementITest (7986 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (7986 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
1 FAILED TEST
I20250114 09:09:53.683430 7606 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5204: suppressed but not reported on 9 messages since previous log ~0 seconds ago
I20250114 09:09:53.683465 7606 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 13 messages since previous log ~5 seconds ago
I20250114 09:09:53.683476 7606 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:643: suppressed but not reported on 13 messages since previous log ~5 seconds ago
I20250114 09:09:53.683519 7606 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/maintenance_manager.cc:345: suppressed but not reported on 99 messages since previous log ~7 seconds ago