Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 19:55:53.443959 5280 test_util.cc:268] Using random seed: -540792224
I20240501 19:55:53.450646 5280 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:55:53.450755 5280 ts_itest-base.cc:113] --------------
I20240501 19:55:53.450825 5280 ts_itest-base.cc:114] 5 tablet servers
I20240501 19:55:53.450860 5280 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:55:53.450873 5280 ts_itest-base.cc:116] --------------
2024-05-01T19:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:53Z Disabled control of system clock
I20240501 19:55:53.459729 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.40.62:37567
--webserver_interface=127.5.40.62
--webserver_port=0
--builtin_ntp_servers=127.5.40.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.40.62:37567
--enable_range_replica_placement=false with env {}
W20240501 19:55:53.564683 5292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:53.564921 5292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:53.564965 5292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:53.567978 5292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:53.568038 5292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:53.568058 5292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:53.568089 5292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:53.571695 5292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.40.62:37567
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.40.62:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.40.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:53.572592 5292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:53.573742 5292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:53.581048 5297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.581462 5300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.582106 5298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:53.582561 5292 server_base.cc:1034] running on GCE node
I20240501 19:55:53.583026 5292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:53.583986 5292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:53.585143 5292 hybrid_clock.cc:648] HybridClock initialized: now 1714593353585127 us; error 29 us; skew 500 ppm
I20240501 19:55:53.586998 5292 webserver.cc:460] Webserver started at http://127.5.40.62:36697/ using document root <none> and password file <none>
I20240501 19:55:53.587554 5292 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:53.587611 5292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:53.587797 5292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:53.589397 5292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6a15c36e43854dffa32f25cbcb5f4a9d"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-bz40"
I20240501 19:55:53.589852 5292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6a15c36e43854dffa32f25cbcb5f4a9d"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-bz40"
I20240501 19:55:53.593235 5292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:53.595345 5306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.596366 5292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:53.596469 5292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6a15c36e43854dffa32f25cbcb5f4a9d"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-bz40"
I20240501 19:55:53.596555 5292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:53.612085 5292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:53.612717 5292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:53.612876 5292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:53.619232 5358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.62:37567 every 8 connection(s)
I20240501 19:55:53.619230 5292 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.62:37567
I20240501 19:55:53.620281 5292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:55:53.622747 5359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:53.624415 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 5292
I20240501 19:55:53.624580 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:55:53.628715 5359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d: Bootstrap starting.
I20240501 19:55:53.631475 5359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:53.632275 5359 log.cc:824] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:53.634223 5359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d: No bootstrap required, opened a new log
I20240501 19:55:53.637374 5359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a15c36e43854dffa32f25cbcb5f4a9d" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37567 } }
I20240501 19:55:53.637616 5359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:53.637681 5359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a15c36e43854dffa32f25cbcb5f4a9d, State: Initialized, Role: FOLLOWER
I20240501 19:55:53.638123 5359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a15c36e43854dffa32f25cbcb5f4a9d" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37567 } }
I20240501 19:55:53.638239 5359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:53.638281 5359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:53.638396 5359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:53.639422 5359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a15c36e43854dffa32f25cbcb5f4a9d" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37567 } }
I20240501 19:55:53.639780 5359 leader_election.cc:302] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a15c36e43854dffa32f25cbcb5f4a9d; no voters:
I20240501 19:55:53.640098 5359 leader_election.cc:288] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:53.640367 5364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:53.640554 5364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [term 1 LEADER]: Becoming Leader. State: Replica: 6a15c36e43854dffa32f25cbcb5f4a9d, State: Running, Role: LEADER
I20240501 19:55:53.640868 5364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a15c36e43854dffa32f25cbcb5f4a9d" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37567 } }
I20240501 19:55:53.641289 5359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:53.642529 5365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a15c36e43854dffa32f25cbcb5f4a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a15c36e43854dffa32f25cbcb5f4a9d" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37567 } } }
I20240501 19:55:53.642551 5366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a15c36e43854dffa32f25cbcb5f4a9d. Latest consensus state: current_term: 1 leader_uuid: "6a15c36e43854dffa32f25cbcb5f4a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a15c36e43854dffa32f25cbcb5f4a9d" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37567 } } }
I20240501 19:55:53.642653 5366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:53.642653 5365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:53.643102 5374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:53.645888 5374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:53.651036 5374 catalog_manager.cc:1347] Generated new cluster ID: d2267b10b9b844869b908d8602c357eb
I20240501 19:55:53.651098 5374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:53.669421 5374 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:53.670722 5374 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:53.684866 5374 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6a15c36e43854dffa32f25cbcb5f4a9d: Generated new TSK 0
I20240501 19:55:53.685734 5374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:53.696143 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.1:0
--local_ip_for_outbound_sockets=127.5.40.1
--webserver_interface=127.5.40.1
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--builtin_ntp_servers=127.5.40.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 19:55:53.803691 5384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:53.803918 5384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:53.803964 5384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:53.806955 5384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:53.807056 5384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.1
I20240501 19:55:53.810601 5384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.40.1
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--never_fsync=true
--heap_profile_path=/tmp/kudu.5384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.1
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:53.811491 5384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:53.812573 5384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:53.820372 5389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.820873 5392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.821554 5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:53.822019 5384 server_base.cc:1034] running on GCE node
I20240501 19:55:53.822463 5384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:53.823151 5384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:53.824324 5384 hybrid_clock.cc:648] HybridClock initialized: now 1714593353824308 us; error 34 us; skew 500 ppm
I20240501 19:55:53.826221 5384 webserver.cc:460] Webserver started at http://127.5.40.1:37253/ using document root <none> and password file <none>
I20240501 19:55:53.826776 5384 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:53.826833 5384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:53.827008 5384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:53.828681 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0597f2422b8643d6b4fd286e711b9ca2"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-bz40"
I20240501 19:55:53.829129 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0597f2422b8643d6b4fd286e711b9ca2"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-bz40"
I20240501 19:55:53.832456 5384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 19:55:53.834573 5398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.835624 5384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:55:53.835726 5384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0597f2422b8643d6b4fd286e711b9ca2"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-bz40"
I20240501 19:55:53.835812 5384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:53.853665 5384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:53.854354 5384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:53.854516 5384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:53.855083 5384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:53.856081 5384 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:53.856132 5384 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.856186 5384 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:53.856213 5384 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:53.864387 5384 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.1:33109
I20240501 19:55:53.864418 5511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.1:33109 every 8 connection(s)
I20240501 19:55:53.865341 5384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:55:53.871671 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 5384
I20240501 19:55:53.871755 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:55:53.873651 5512 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37567
I20240501 19:55:53.873811 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.2:0
--local_ip_for_outbound_sockets=127.5.40.2
--webserver_interface=127.5.40.2
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--builtin_ntp_servers=127.5.40.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 19:55:53.873857 5512 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:53.874363 5512 heartbeater.cc:511] Master 127.5.40.62:37567 requested a full tablet report, sending...
I20240501 19:55:53.875908 5323 ts_manager.cc:194] Registered new tserver with Master: 0597f2422b8643d6b4fd286e711b9ca2 (127.5.40.1:33109)
I20240501 19:55:53.877476 5323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.1:56731
W20240501 19:55:53.977032 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:53.977247 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:53.977296 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:53.980454 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:53.980558 5516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.2
I20240501 19:55:53.984138 5516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.40.2
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--never_fsync=true
--heap_profile_path=/tmp/kudu.5516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.2
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:53.985005 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:53.986078 5516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:53.993886 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.994386 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:53.994853 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:53.995400 5516 server_base.cc:1034] running on GCE node
I20240501 19:55:53.995770 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:53.996253 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:53.997391 5516 hybrid_clock.cc:648] HybridClock initialized: now 1714593353997374 us; error 39 us; skew 500 ppm
I20240501 19:55:53.998992 5516 webserver.cc:460] Webserver started at http://127.5.40.2:32811/ using document root <none> and password file <none>
I20240501 19:55:53.999539 5516 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:53.999600 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:53.999763 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:54.001370 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "382fcb688ce845298f1710424e01de25"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-bz40"
I20240501 19:55:54.001835 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "382fcb688ce845298f1710424e01de25"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-bz40"
I20240501 19:55:54.004957 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:54.006948 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.007928 5516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:54.008025 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "382fcb688ce845298f1710424e01de25"
format_stamp: "Formatted at 2024-05-01 19:55:53 on dist-test-slave-bz40"
I20240501 19:55:54.008102 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:54.022037 5516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:54.022703 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:54.022857 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:54.023407 5516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:54.024318 5516 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:54.024366 5516 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.024420 5516 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:54.024451 5516 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.032860 5516 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.2:33721
I20240501 19:55:54.032898 5643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.2:33721 every 8 connection(s)
I20240501 19:55:54.033831 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:55:54.039222 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 5516
I20240501 19:55:54.039341 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:55:54.041692 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.3:0
--local_ip_for_outbound_sockets=127.5.40.3
--webserver_interface=127.5.40.3
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--builtin_ntp_servers=127.5.40.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 19:55:54.042769 5644 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37567
I20240501 19:55:54.043026 5644 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:54.043622 5644 heartbeater.cc:511] Master 127.5.40.62:37567 requested a full tablet report, sending...
I20240501 19:55:54.044646 5323 ts_manager.cc:194] Registered new tserver with Master: 382fcb688ce845298f1710424e01de25 (127.5.40.2:33721)
I20240501 19:55:54.045410 5323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.2:35475
W20240501 19:55:54.145617 5647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:54.145836 5647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:54.145877 5647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:54.148973 5647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:54.149086 5647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.3
I20240501 19:55:54.152773 5647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.40.3
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--never_fsync=true
--heap_profile_path=/tmp/kudu.5647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.3
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:54.153633 5647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:54.154706 5647 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:54.162451 5653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.162854 5655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.163481 5652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:54.163861 5647 server_base.cc:1034] running on GCE node
I20240501 19:55:54.164243 5647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:54.164762 5647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:54.165926 5647 hybrid_clock.cc:648] HybridClock initialized: now 1714593354165912 us; error 24 us; skew 500 ppm
I20240501 19:55:54.167703 5647 webserver.cc:460] Webserver started at http://127.5.40.3:46723/ using document root <none> and password file <none>
I20240501 19:55:54.168224 5647 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:54.168275 5647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:54.168432 5647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:54.170044 5647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "72af66cb81d14237978e5ff11a972985"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-bz40"
I20240501 19:55:54.170441 5647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "72af66cb81d14237978e5ff11a972985"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-bz40"
I20240501 19:55:54.173542 5647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20240501 19:55:54.175645 5661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.176640 5647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:54.176736 5647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "72af66cb81d14237978e5ff11a972985"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-bz40"
I20240501 19:55:54.176812 5647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:54.193770 5647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:54.194351 5647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:54.194484 5647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:54.194950 5647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:54.195817 5647 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:54.195863 5647 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.195900 5647 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:54.195920 5647 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.203727 5647 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.3:35917
I20240501 19:55:54.203758 5774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.3:35917 every 8 connection(s)
I20240501 19:55:54.204624 5647 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:55:54.206913 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 5647
I20240501 19:55:54.207015 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:55:54.208905 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.4:0
--local_ip_for_outbound_sockets=127.5.40.4
--webserver_interface=127.5.40.4
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--builtin_ntp_servers=127.5.40.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 19:55:54.212843 5775 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37567
I20240501 19:55:54.213054 5775 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:54.213522 5775 heartbeater.cc:511] Master 127.5.40.62:37567 requested a full tablet report, sending...
I20240501 19:55:54.214403 5323 ts_manager.cc:194] Registered new tserver with Master: 72af66cb81d14237978e5ff11a972985 (127.5.40.3:35917)
I20240501 19:55:54.214924 5323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.3:59683
W20240501 19:55:54.312067 5778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:54.312304 5778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:54.312368 5778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:54.315423 5778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:54.315580 5778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.4
I20240501 19:55:54.319178 5778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.40.4
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--never_fsync=true
--heap_profile_path=/tmp/kudu.5778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.4
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:54.320096 5778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:54.321210 5778 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:54.329089 5785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.329492 5784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.330107 5787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:54.330600 5778 server_base.cc:1034] running on GCE node
I20240501 19:55:54.330968 5778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:54.331511 5778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:54.332656 5778 hybrid_clock.cc:648] HybridClock initialized: now 1714593354332643 us; error 25 us; skew 500 ppm
I20240501 19:55:54.334319 5778 webserver.cc:460] Webserver started at http://127.5.40.4:40313/ using document root <none> and password file <none>
I20240501 19:55:54.334846 5778 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:54.334903 5778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:54.335076 5778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:54.336740 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "cfbb24648f934541b5020f1d939bed16"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-bz40"
I20240501 19:55:54.337209 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "cfbb24648f934541b5020f1d939bed16"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-bz40"
I20240501 19:55:54.340523 5778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:54.342748 5793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.343755 5778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:55:54.343855 5778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "cfbb24648f934541b5020f1d939bed16"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-bz40"
I20240501 19:55:54.343945 5778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:54.365612 5778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:54.366331 5778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:54.366501 5778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:54.367046 5778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:54.367986 5778 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:54.368034 5778 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.368086 5778 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:54.368115 5778 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.376243 5778 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.4:41929
I20240501 19:55:54.376269 5907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.4:41929 every 8 connection(s)
I20240501 19:55:54.377171 5778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:55:54.384802 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 5778
I20240501 19:55:54.384903 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:55:54.385459 5908 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37567
I20240501 19:55:54.385679 5908 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:54.386186 5908 heartbeater.cc:511] Master 127.5.40.62:37567 requested a full tablet report, sending...
I20240501 19:55:54.386790 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.5:0
--local_ip_for_outbound_sockets=127.5.40.5
--webserver_interface=127.5.40.5
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--builtin_ntp_servers=127.5.40.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 19:55:54.387167 5323 ts_manager.cc:194] Registered new tserver with Master: cfbb24648f934541b5020f1d939bed16 (127.5.40.4:41929)
I20240501 19:55:54.387849 5323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.4:55401
W20240501 19:55:54.490396 5911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:54.490600 5911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:54.490657 5911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:54.493633 5911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:54.493752 5911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.5
I20240501 19:55:54.497292 5911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.40.5
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37567
--never_fsync=true
--heap_profile_path=/tmp/kudu.5911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.5
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:54.498143 5911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:54.499215 5911 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:54.507062 5919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.507500 5916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:54.507947 5917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:54.508500 5911 server_base.cc:1034] running on GCE node
I20240501 19:55:54.508802 5911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:54.509287 5911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:54.510442 5911 hybrid_clock.cc:648] HybridClock initialized: now 1714593354510428 us; error 27 us; skew 500 ppm
I20240501 19:55:54.512238 5911 webserver.cc:460] Webserver started at http://127.5.40.5:33777/ using document root <none> and password file <none>
I20240501 19:55:54.512710 5911 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:54.512758 5911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:54.512914 5911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:54.514394 5911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "69c69c39965741b7b95da4b598837d9f"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-bz40"
I20240501 19:55:54.514804 5911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "69c69c39965741b7b95da4b598837d9f"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-bz40"
I20240501 19:55:54.517786 5911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 19:55:54.519752 5925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.520678 5911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:54.520776 5911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "69c69c39965741b7b95da4b598837d9f"
format_stamp: "Formatted at 2024-05-01 19:55:54 on dist-test-slave-bz40"
I20240501 19:55:54.520862 5911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:54.538028 5911 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:54.538645 5911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:54.538794 5911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:54.539336 5911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:54.540241 5911 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:54.540287 5911 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.540338 5911 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:54.540365 5911 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:54.548573 5911 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.5:38315
I20240501 19:55:54.548607 6040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.5:38315 every 8 connection(s)
I20240501 19:55:54.549501 5911 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:55:54.552575 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 5911
I20240501 19:55:54.552672 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:55:54.557708 6041 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37567
I20240501 19:55:54.557909 6041 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:54.558379 6041 heartbeater.cc:511] Master 127.5.40.62:37567 requested a full tablet report, sending...
I20240501 19:55:54.559247 5323 ts_manager.cc:194] Registered new tserver with Master: 69c69c39965741b7b95da4b598837d9f (127.5.40.5:38315)
I20240501 19:55:54.559783 5323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.5:37845
I20240501 19:55:54.566650 5280 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:55:54.577483 5323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50940:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:55:54.594410 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 64c0f4792ba84216991add475dee2473 (DEFAULT_TABLE table=TestTable [id=eb25308e2f25478f842da9d440d677cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:54.594410 5841 tablet_service.cc:1467] Processing CreateTablet for tablet 64c0f4792ba84216991add475dee2473 (DEFAULT_TABLE table=TestTable [id=eb25308e2f25478f842da9d440d677cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:54.594383 5709 tablet_service.cc:1467] Processing CreateTablet for tablet 64c0f4792ba84216991add475dee2473 (DEFAULT_TABLE table=TestTable [id=eb25308e2f25478f842da9d440d677cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:54.595316 5709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c0f4792ba84216991add475dee2473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:54.595331 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c0f4792ba84216991add475dee2473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:54.595326 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c0f4792ba84216991add475dee2473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:54.600149 6058 tablet_bootstrap.cc:492] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985: Bootstrap starting.
I20240501 19:55:54.601419 6060 tablet_bootstrap.cc:492] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16: Bootstrap starting.
I20240501 19:55:54.602377 6058 tablet_bootstrap.cc:654] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:54.603046 6060 tablet_bootstrap.cc:654] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:54.603132 6058 log.cc:824] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:54.603399 6059 tablet_bootstrap.cc:492] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f: Bootstrap starting.
I20240501 19:55:54.604599 6058 tablet_bootstrap.cc:492] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985: No bootstrap required, opened a new log
I20240501 19:55:54.604753 6058 ts_tablet_manager.cc:1367] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:55:54.604894 6059 tablet_bootstrap.cc:654] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:54.605473 6059 log.cc:824] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:54.606387 6060 log.cc:824] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:54.607136 6059 tablet_bootstrap.cc:492] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f: No bootstrap required, opened a new log
I20240501 19:55:54.607308 6059 ts_tablet_manager.cc:1367] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:55:54.607797 6060 tablet_bootstrap.cc:492] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16: No bootstrap required, opened a new log
I20240501 19:55:54.607959 6060 ts_tablet_manager.cc:1367] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 19:55:54.608052 6058 raft_consensus.cc:348] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } }
I20240501 19:55:54.608376 6058 raft_consensus.cc:374] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:54.608479 6058 raft_consensus.cc:729] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72af66cb81d14237978e5ff11a972985, State: Initialized, Role: FOLLOWER
I20240501 19:55:54.609037 6058 consensus_queue.cc:260] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } }
I20240501 19:55:54.609712 5775 heartbeater.cc:503] Master 127.5.40.62:37567 was elected leader, sending a full tablet report...
I20240501 19:55:54.610925 6059 raft_consensus.cc:348] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } }
I20240501 19:55:54.611124 6059 raft_consensus.cc:374] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:54.611171 6059 raft_consensus.cc:729] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c69c39965741b7b95da4b598837d9f, State: Initialized, Role: FOLLOWER
I20240501 19:55:54.611056 6060 raft_consensus.cc:348] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } }
I20240501 19:55:54.611244 6058 ts_tablet_manager.cc:1398] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20240501 19:55:54.611282 6060 raft_consensus.cc:374] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:54.611330 6060 raft_consensus.cc:729] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfbb24648f934541b5020f1d939bed16, State: Initialized, Role: FOLLOWER
I20240501 19:55:54.611613 6059 consensus_queue.cc:260] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } }
I20240501 19:55:54.611862 6060 consensus_queue.cc:260] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } }
I20240501 19:55:54.612318 6041 heartbeater.cc:503] Master 127.5.40.62:37567 was elected leader, sending a full tablet report...
I20240501 19:55:54.612469 6059 ts_tablet_manager.cc:1398] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 19:55:54.612695 6060 ts_tablet_manager.cc:1398] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:55:54.613018 5908 heartbeater.cc:503] Master 127.5.40.62:37567 was elected leader, sending a full tablet report...
W20240501 19:55:54.628037 5909 tablet.cc:2369] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:55:54.634176 6064 raft_consensus.cc:482] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:55:54.634370 6064 raft_consensus.cc:504] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } }
I20240501 19:55:54.635510 6064 leader_election.cc:288] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfbb24648f934541b5020f1d939bed16 (127.5.40.4:41929), 69c69c39965741b7b95da4b598837d9f (127.5.40.5:38315)
I20240501 19:55:54.639652 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c0f4792ba84216991add475dee2473" candidate_uuid: "72af66cb81d14237978e5ff11a972985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfbb24648f934541b5020f1d939bed16" is_pre_election: true
I20240501 19:55:54.639652 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c0f4792ba84216991add475dee2473" candidate_uuid: "72af66cb81d14237978e5ff11a972985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69c69c39965741b7b95da4b598837d9f" is_pre_election: true
I20240501 19:55:54.639953 5862 raft_consensus.cc:2454] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72af66cb81d14237978e5ff11a972985 in term 0.
I20240501 19:55:54.639953 5994 raft_consensus.cc:2454] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72af66cb81d14237978e5ff11a972985 in term 0.
I20240501 19:55:54.640448 5663 leader_election.cc:302] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72af66cb81d14237978e5ff11a972985, cfbb24648f934541b5020f1d939bed16; no voters:
I20240501 19:55:54.640691 6064 raft_consensus.cc:2781] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:55:54.640759 6064 raft_consensus.cc:482] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:55:54.640789 6064 raft_consensus.cc:3037] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:54.641672 6064 raft_consensus.cc:504] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } }
I20240501 19:55:54.642014 6064 leader_election.cc:288] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [CANDIDATE]: Term 1 election: Requested vote from peers cfbb24648f934541b5020f1d939bed16 (127.5.40.4:41929), 69c69c39965741b7b95da4b598837d9f (127.5.40.5:38315)
I20240501 19:55:54.642336 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c0f4792ba84216991add475dee2473" candidate_uuid: "72af66cb81d14237978e5ff11a972985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfbb24648f934541b5020f1d939bed16"
I20240501 19:55:54.642350 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c0f4792ba84216991add475dee2473" candidate_uuid: "72af66cb81d14237978e5ff11a972985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69c69c39965741b7b95da4b598837d9f"
I20240501 19:55:54.642437 5862 raft_consensus.cc:3037] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:54.642540 5994 raft_consensus.cc:3037] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:54.643282 5862 raft_consensus.cc:2454] T 64c0f4792ba84216991add475dee2473 P cfbb24648f934541b5020f1d939bed16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72af66cb81d14237978e5ff11a972985 in term 1.
I20240501 19:55:54.643520 5994 raft_consensus.cc:2454] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72af66cb81d14237978e5ff11a972985 in term 1.
I20240501 19:55:54.643587 5663 leader_election.cc:302] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72af66cb81d14237978e5ff11a972985, cfbb24648f934541b5020f1d939bed16; no voters:
I20240501 19:55:54.643750 6064 raft_consensus.cc:2781] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:54.643970 6064 raft_consensus.cc:686] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [term 1 LEADER]: Becoming Leader. State: Replica: 72af66cb81d14237978e5ff11a972985, State: Running, Role: LEADER
I20240501 19:55:54.644290 6064 consensus_queue.cc:237] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } }
I20240501 19:55:54.647285 5322 catalog_manager.cc:5484] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72af66cb81d14237978e5ff11a972985 (127.5.40.3). New cstate: current_term: 1 leader_uuid: "72af66cb81d14237978e5ff11a972985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72af66cb81d14237978e5ff11a972985" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 35917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 } health_report { overall_health: UNKNOWN } } }
I20240501 19:55:54.667619 5280 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:55:54.669248 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0597f2422b8643d6b4fd286e711b9ca2 to finish bootstrapping
I20240501 19:55:54.674970 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 382fcb688ce845298f1710424e01de25 to finish bootstrapping
I20240501 19:55:54.680173 5280 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 72af66cb81d14237978e5ff11a972985 to finish bootstrapping
I20240501 19:55:54.684698 5280 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cfbb24648f934541b5020f1d939bed16 to finish bootstrapping
I20240501 19:55:54.689167 5280 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 69c69c39965741b7b95da4b598837d9f to finish bootstrapping
W20240501 19:55:54.705576 5776 tablet.cc:2369] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:55:54.800293 6042 tablet.cc:2369] T 64c0f4792ba84216991add475dee2473 P 69c69c39965741b7b95da4b598837d9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:55:54.867991 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.873703 5578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.876121 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.876698 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:54.877781 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:54.880089 5512 heartbeater.cc:503] Master 127.5.40.62:37567 was elected leader, sending a full tablet report...
I20240501 19:55:55.042186 6064 consensus_queue.cc:1035] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69c69c39965741b7b95da4b598837d9f" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:55:55.047346 5644 heartbeater.cc:503] Master 127.5.40.62:37567 was elected leader, sending a full tablet report...
I20240501 19:55:55.048489 6064 consensus_queue.cc:1035] T 64c0f4792ba84216991add475dee2473 P 72af66cb81d14237978e5ff11a972985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfbb24648f934541b5020f1d939bed16" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 41929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:55:55.105976 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.105988 5578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.107908 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.111336 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.120182 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:55.329388 5578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.330065 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.331661 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.332322 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.338902 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:55.420109 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:55.420142 5578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.421193 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.421412 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.424439 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.523905 5578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.524255 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.525630 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:55.525863 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.526624 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.703063 5322 ts_manager.cc:292] Set tserver state for 0597f2422b8643d6b4fd286e711b9ca2 to MAINTENANCE_MODE
I20240501 19:55:55.903728 5578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.903890 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.905166 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:55.905375 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:55.906860 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.001699 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.002012 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.004005 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.012050 5578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.022855 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.137740 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.138494 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.140125 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.153730 5578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.155128 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.233300 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.236364 5578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.236665 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.238130 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.243752 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.272673 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 5516
I20240501 19:55:56.463783 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.464000 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.465269 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.467723 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.543220 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.544241 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.546343 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.551599 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.652899 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.655002 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.656155 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.656978 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.730698 5446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.731459 5709 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:56.731492 5974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.732307 5841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:56.765014 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 5384
I20240501 19:55:56.769906 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 5647
I20240501 19:55:56.774456 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 5778
I20240501 19:55:56.779433 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 5911
I20240501 19:55:56.784143 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 5292
2024-05-01T19:55:56Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (3362 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (3362 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240501 19:55:56.805366 5280 test_util.cc:268] Using random seed: -537430802
I20240501 19:55:56.806260 5280 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:55:56.806310 5280 ts_itest-base.cc:113] --------------
I20240501 19:55:56.806326 5280 ts_itest-base.cc:114] 5 tablet servers
I20240501 19:55:56.806340 5280 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:55:56.806352 5280 ts_itest-base.cc:116] --------------
2024-05-01T19:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:56Z Disabled control of system clock
I20240501 19:55:56.814069 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.40.62:37895
--webserver_interface=127.5.40.62
--webserver_port=0
--builtin_ntp_servers=127.5.40.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.40.62:37895
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 19:55:56.920610 6510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:56.920832 6510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:56.920872 6510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:56.924062 6510 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:56.924129 6510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:56.924151 6510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:56.924185 6510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:56.924214 6510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:56.927963 6510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.40.62:37895
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.40.62:37895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.40.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:56.928838 6510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:56.929967 6510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:56.937327 6515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:56.937615 6516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:56.938197 6518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:56.938716 6510 server_base.cc:1034] running on GCE node
I20240501 19:55:56.939237 6510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:56.940145 6510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:56.941291 6510 hybrid_clock.cc:648] HybridClock initialized: now 1714593356941280 us; error 26 us; skew 500 ppm
I20240501 19:55:56.943188 6510 webserver.cc:460] Webserver started at http://127.5.40.62:40587/ using document root <none> and password file <none>
I20240501 19:55:56.943753 6510 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:56.943809 6510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:56.943993 6510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:56.945632 6510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25"
format_stamp: "Formatted at 2024-05-01 19:55:56 on dist-test-slave-bz40"
I20240501 19:55:56.946096 6510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25"
format_stamp: "Formatted at 2024-05-01 19:55:56 on dist-test-slave-bz40"
I20240501 19:55:56.949301 6510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:56.951342 6524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:56.952360 6510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:56.952463 6510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25"
format_stamp: "Formatted at 2024-05-01 19:55:56 on dist-test-slave-bz40"
I20240501 19:55:56.952551 6510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:56.985352 6510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:56.986189 6510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:56.986353 6510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:56.992806 6576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.62:37895 every 8 connection(s)
I20240501 19:55:56.992790 6510 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.62:37895
I20240501 19:55:56.993796 6510 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:55:56.996340 6577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:56.998497 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 6510
I20240501 19:55:56.998617 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:55:57.001777 6577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25: Bootstrap starting.
I20240501 19:55:57.003854 6577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:57.004616 6577 log.cc:824] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:57.006474 6577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25: No bootstrap required, opened a new log
I20240501 19:55:57.009136 6577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37895 } }
I20240501 19:55:57.009312 6577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:57.009344 6577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b67e1ea7e164e66acf0b7ca94c3ed25, State: Initialized, Role: FOLLOWER
I20240501 19:55:57.009696 6577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37895 } }
I20240501 19:55:57.009794 6577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:57.009837 6577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:57.009912 6577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:57.070621 6577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37895 } }
I20240501 19:55:57.071089 6577 leader_election.cc:302] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b67e1ea7e164e66acf0b7ca94c3ed25; no voters:
I20240501 19:55:57.071422 6577 leader_election.cc:288] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:57.071609 6582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:57.072115 6582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [term 1 LEADER]: Becoming Leader. State: Replica: 4b67e1ea7e164e66acf0b7ca94c3ed25, State: Running, Role: LEADER
I20240501 19:55:57.072479 6582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37895 } }
I20240501 19:55:57.072726 6577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:57.074060 6584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b67e1ea7e164e66acf0b7ca94c3ed25. Latest consensus state: current_term: 1 leader_uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37895 } } }
I20240501 19:55:57.074188 6584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:57.074491 6583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b67e1ea7e164e66acf0b7ca94c3ed25" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 37895 } } }
I20240501 19:55:57.074591 6583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25 [sys.catalog]: This master's current role is: LEADER
W20240501 19:55:57.077567 6597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:55:57.077636 6597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:55:57.077725 6592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:57.078650 6592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:57.083868 6592 catalog_manager.cc:1347] Generated new cluster ID: 37739681b6994babb0154e6e06ea4eda
I20240501 19:55:57.083949 6592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:57.092109 6592 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:57.093201 6592 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:57.111608 6592 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4b67e1ea7e164e66acf0b7ca94c3ed25: Generated new TSK 0
I20240501 19:55:57.112335 6592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:57.121932 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.1:0
--local_ip_for_outbound_sockets=127.5.40.1
--webserver_interface=127.5.40.1
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--builtin_ntp_servers=127.5.40.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:55:57.247268 6601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.247546 6601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.247610 6601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.250766 6601 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:57.250841 6601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.250921 6601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.1
I20240501 19:55:57.254721 6601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.40.1
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--never_fsync=true
--heap_profile_path=/tmp/kudu.6601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.1
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.255654 6601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.256830 6601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.266216 6606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.266639 6609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.267112 6607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.267638 6601 server_base.cc:1034] running on GCE node
I20240501 19:55:57.268007 6601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.268568 6601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.269713 6601 hybrid_clock.cc:648] HybridClock initialized: now 1714593357269699 us; error 29 us; skew 500 ppm
I20240501 19:55:57.271636 6601 webserver.cc:460] Webserver started at http://127.5.40.1:37617/ using document root <none> and password file <none>
I20240501 19:55:57.272213 6601 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.272282 6601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.272454 6601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.274215 6601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.274704 6601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.278101 6601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 19:55:57.280360 6615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.281392 6601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:55:57.281526 6601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.281608 6601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.309670 6601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.310376 6601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.310547 6601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.311106 6601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.312059 6601 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.312108 6601 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.312162 6601 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.312191 6601 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.320794 6601 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.1:46275
I20240501 19:55:57.320818 6728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.1:46275 every 8 connection(s)
I20240501 19:55:57.321811 6601 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:55:57.328953 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 6601
I20240501 19:55:57.329067 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:55:57.330854 6729 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37895
I20240501 19:55:57.331072 6729 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.331306 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.2:0
--local_ip_for_outbound_sockets=127.5.40.2
--webserver_interface=127.5.40.2
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--builtin_ntp_servers=127.5.40.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:55:57.331631 6729 heartbeater.cc:511] Master 127.5.40.62:37895 requested a full tablet report, sending...
I20240501 19:55:57.333086 6541 ts_manager.cc:194] Registered new tserver with Master: e75dbd1c455a4dfd9a041ee3d5cac0c3 (127.5.40.1:46275)
I20240501 19:55:57.334266 6541 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.1:54613
W20240501 19:55:57.437110 6732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.437325 6732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.437366 6732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.440378 6732 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:57.440440 6732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.440512 6732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.2
I20240501 19:55:57.444051 6732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.40.2
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--never_fsync=true
--heap_profile_path=/tmp/kudu.6732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.2
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.444909 6732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.445981 6732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.454236 6738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.454478 6737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.455005 6740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.455582 6732 server_base.cc:1034] running on GCE node
I20240501 19:55:57.456005 6732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.456580 6732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.457749 6732 hybrid_clock.cc:648] HybridClock initialized: now 1714593357457732 us; error 28 us; skew 500 ppm
I20240501 19:55:57.459566 6732 webserver.cc:460] Webserver started at http://127.5.40.2:35175/ using document root <none> and password file <none>
I20240501 19:55:57.460094 6732 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.460147 6732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.460320 6732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.461988 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "684d010e5b4c428691cd8936e85a77ec"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.462512 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "684d010e5b4c428691cd8936e85a77ec"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.465672 6732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 19:55:57.467794 6746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.468766 6732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.468884 6732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "684d010e5b4c428691cd8936e85a77ec"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.468962 6732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.485950 6732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.486547 6732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.486709 6732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.487251 6732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.488139 6732 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.488184 6732 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.488237 6732 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.488269 6732 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.496119 6732 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.2:40497
I20240501 19:55:57.496356 6859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.2:40497 every 8 connection(s)
I20240501 19:55:57.497227 6732 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:55:57.505525 6860 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37895
I20240501 19:55:57.505753 6860 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.506345 6860 heartbeater.cc:511] Master 127.5.40.62:37895 requested a full tablet report, sending...
I20240501 19:55:57.507038 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 6732
I20240501 19:55:57.507133 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:55:57.507408 6541 ts_manager.cc:194] Registered new tserver with Master: 684d010e5b4c428691cd8936e85a77ec (127.5.40.2:40497)
I20240501 19:55:57.508105 6541 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.2:58245
I20240501 19:55:57.509399 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.3:0
--local_ip_for_outbound_sockets=127.5.40.3
--webserver_interface=127.5.40.3
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--builtin_ntp_servers=127.5.40.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:55:57.614670 6863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.614898 6863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.614943 6863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.617965 6863 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:57.618034 6863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.618101 6863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.3
I20240501 19:55:57.621719 6863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.40.3
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--never_fsync=true
--heap_profile_path=/tmp/kudu.6863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.3
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.622637 6863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.623759 6863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.632242 6869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.632814 6868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.633327 6871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.634011 6863 server_base.cc:1034] running on GCE node
I20240501 19:55:57.634425 6863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.634991 6863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.636193 6863 hybrid_clock.cc:648] HybridClock initialized: now 1714593357636174 us; error 25 us; skew 500 ppm
I20240501 19:55:57.638295 6863 webserver.cc:460] Webserver started at http://127.5.40.3:36853/ using document root <none> and password file <none>
I20240501 19:55:57.638856 6863 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.638917 6863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.639091 6863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.641072 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "61e6abec1ce444cdb5a4b05a094c1300"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.641598 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "61e6abec1ce444cdb5a4b05a094c1300"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.645053 6863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 19:55:57.647289 6877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.648351 6863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.648455 6863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "61e6abec1ce444cdb5a4b05a094c1300"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.648540 6863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.683192 6863 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.683856 6863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.684015 6863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.684542 6863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.685493 6863 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.685540 6863 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.685592 6863 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.685621 6863 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.693524 6863 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.3:33265
I20240501 19:55:57.693804 6992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.3:33265 every 8 connection(s)
I20240501 19:55:57.696152 6863 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:55:57.705026 6993 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37895
I20240501 19:55:57.705232 6993 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.705637 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 6863
I20240501 19:55:57.705745 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:55:57.705749 6993 heartbeater.cc:511] Master 127.5.40.62:37895 requested a full tablet report, sending...
I20240501 19:55:57.706799 6540 ts_manager.cc:194] Registered new tserver with Master: 61e6abec1ce444cdb5a4b05a094c1300 (127.5.40.3:33265)
I20240501 19:55:57.707491 6540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.3:32943
I20240501 19:55:57.707908 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.4:0
--local_ip_for_outbound_sockets=127.5.40.4
--webserver_interface=127.5.40.4
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--builtin_ntp_servers=127.5.40.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:55:57.812211 6996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.812439 6996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.812502 6996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.815501 6996 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:57.815569 6996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.815655 6996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.4
I20240501 19:55:57.819180 6996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.40.4
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--never_fsync=true
--heap_profile_path=/tmp/kudu.6996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.4
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.820081 6996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.821177 6996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.829293 7001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.829712 7004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.830436 7002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.830906 6996 server_base.cc:1034] running on GCE node
I20240501 19:55:57.831254 6996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.831846 6996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.832996 6996 hybrid_clock.cc:648] HybridClock initialized: now 1714593357832985 us; error 32 us; skew 500 ppm
I20240501 19:55:57.840023 6996 webserver.cc:460] Webserver started at http://127.5.40.4:38761/ using document root <none> and password file <none>
I20240501 19:55:57.840581 6996 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.840641 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.840809 6996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.842504 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f4d410a8669c48bea56dae6c4070858f"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.842947 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f4d410a8669c48bea56dae6c4070858f"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.846120 6996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:57.848206 7010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.849148 6996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.849253 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f4d410a8669c48bea56dae6c4070858f"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-bz40"
I20240501 19:55:57.849337 6996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.867354 6996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.867987 6996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.868144 6996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.868671 6996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.869544 6996 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.869591 6996 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.869642 6996 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.869670 6996 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.877964 6996 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.4:45103
I20240501 19:55:57.877979 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.4:45103 every 8 connection(s)
I20240501 19:55:57.878890 6996 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:55:57.882905 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 6996
I20240501 19:55:57.883028 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:55:57.885362 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.5:0
--local_ip_for_outbound_sockets=127.5.40.5
--webserver_interface=127.5.40.5
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--builtin_ntp_servers=127.5.40.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:55:57.888244 7125 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37895
I20240501 19:55:57.888463 7125 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.889008 7125 heartbeater.cc:511] Master 127.5.40.62:37895 requested a full tablet report, sending...
I20240501 19:55:57.890074 6540 ts_manager.cc:194] Registered new tserver with Master: f4d410a8669c48bea56dae6c4070858f (127.5.40.4:45103)
I20240501 19:55:57.890815 6540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.4:57419
W20240501 19:55:57.991336 7128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.991586 7128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.991632 7128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.994647 7128 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:57.994714 7128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.994781 7128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.5
I20240501 19:55:57.998389 7128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.40.5
--webserver_port=0
--tserver_master_addrs=127.5.40.62:37895
--never_fsync=true
--heap_profile_path=/tmp/kudu.7128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.5
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.999248 7128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.000339 7128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.008533 7137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.008915 7134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.009394 7135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.009997 7128 server_base.cc:1034] running on GCE node
I20240501 19:55:58.010378 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.010915 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.012059 7128 hybrid_clock.cc:648] HybridClock initialized: now 1714593358012041 us; error 42 us; skew 500 ppm
I20240501 19:55:58.013897 7128 webserver.cc:460] Webserver started at http://127.5.40.5:37267/ using document root <none> and password file <none>
I20240501 19:55:58.014421 7128 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.014487 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.014678 7128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.016379 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1f04154b01f44df6b969d7abc20a41ed"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-bz40"
I20240501 19:55:58.016846 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1f04154b01f44df6b969d7abc20a41ed"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-bz40"
I20240501 19:55:58.020176 7128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:58.022325 7143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.023332 7128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.023475 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1f04154b01f44df6b969d7abc20a41ed"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-bz40"
I20240501 19:55:58.023563 7128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.039274 7128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.039921 7128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.040078 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.040608 7128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.041556 7128 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.041615 7128 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.041669 7128 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.041703 7128 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.049599 7128 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.5:38413
I20240501 19:55:58.049628 7256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.5:38413 every 8 connection(s)
I20240501 19:55:58.050500 7128 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:55:58.050997 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 7128
I20240501 19:55:58.051117 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:55:58.058943 7257 heartbeater.cc:347] Connected to a master server at 127.5.40.62:37895
I20240501 19:55:58.059161 7257 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.059712 7257 heartbeater.cc:511] Master 127.5.40.62:37895 requested a full tablet report, sending...
I20240501 19:55:58.060678 6540 ts_manager.cc:194] Registered new tserver with Master: 1f04154b01f44df6b969d7abc20a41ed (127.5.40.5:38413)
I20240501 19:55:58.061208 6540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.5:44747
I20240501 19:55:58.065106 5280 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:55:58.073194 6540 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60602:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:55:58.090458 6794 tablet_service.cc:1467] Processing CreateTablet for tablet f894bc2ae25e4fe9ab52c11f64094b2c (DEFAULT_TABLE table=TestTable [id=e8eafa1df0b842c58dfeeba9fa969a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:58.090444 6663 tablet_service.cc:1467] Processing CreateTablet for tablet f894bc2ae25e4fe9ab52c11f64094b2c (DEFAULT_TABLE table=TestTable [id=e8eafa1df0b842c58dfeeba9fa969a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:58.091475 6663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f894bc2ae25e4fe9ab52c11f64094b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:58.091403 7191 tablet_service.cc:1467] Processing CreateTablet for tablet f894bc2ae25e4fe9ab52c11f64094b2c (DEFAULT_TABLE table=TestTable [id=e8eafa1df0b842c58dfeeba9fa969a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:55:58.092278 7191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f894bc2ae25e4fe9ab52c11f64094b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:58.094058 6794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f894bc2ae25e4fe9ab52c11f64094b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:58.099871 7276 tablet_bootstrap.cc:492] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec: Bootstrap starting.
I20240501 19:55:58.100268 7275 tablet_bootstrap.cc:492] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed: Bootstrap starting.
I20240501 19:55:58.101714 7276 tablet_bootstrap.cc:654] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:58.101858 7275 tablet_bootstrap.cc:654] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:58.102365 7276 log.cc:824] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:58.102430 7275 log.cc:824] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:58.103109 7274 tablet_bootstrap.cc:492] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3: Bootstrap starting.
I20240501 19:55:58.104099 7276 tablet_bootstrap.cc:492] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec: No bootstrap required, opened a new log
I20240501 19:55:58.104281 7276 ts_tablet_manager.cc:1367] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:55:58.105157 7274 tablet_bootstrap.cc:654] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:58.105906 7274 log.cc:824] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:58.107311 7274 tablet_bootstrap.cc:492] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3: No bootstrap required, opened a new log
I20240501 19:55:58.107370 7275 tablet_bootstrap.cc:492] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed: No bootstrap required, opened a new log
I20240501 19:55:58.107569 7275 ts_tablet_manager.cc:1367] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 19:55:58.107564 7274 ts_tablet_manager.cc:1367] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:55:58.107491 7276 raft_consensus.cc:348] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } }
I20240501 19:55:58.107815 7276 raft_consensus.cc:374] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:58.107920 7276 raft_consensus.cc:729] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684d010e5b4c428691cd8936e85a77ec, State: Initialized, Role: FOLLOWER
I20240501 19:55:58.108496 7276 consensus_queue.cc:260] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } }
I20240501 19:55:58.109154 6860 heartbeater.cc:503] Master 127.5.40.62:37895 was elected leader, sending a full tablet report...
I20240501 19:55:58.110286 7274 raft_consensus.cc:348] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } }
I20240501 19:55:58.110534 7274 raft_consensus.cc:374] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:58.110632 7274 raft_consensus.cc:729] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75dbd1c455a4dfd9a041ee3d5cac0c3, State: Initialized, Role: FOLLOWER
I20240501 19:55:58.110694 7276 ts_tablet_manager.cc:1398] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:55:58.111122 7274 consensus_queue.cc:260] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } }
I20240501 19:55:58.111114 7275 raft_consensus.cc:348] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } }
I20240501 19:55:58.111259 7275 raft_consensus.cc:374] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:58.111299 7275 raft_consensus.cc:729] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f04154b01f44df6b969d7abc20a41ed, State: Initialized, Role: FOLLOWER
I20240501 19:55:58.111796 6729 heartbeater.cc:503] Master 127.5.40.62:37895 was elected leader, sending a full tablet report...
I20240501 19:55:58.111757 7275 consensus_queue.cc:260] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } }
I20240501 19:55:58.111984 7274 ts_tablet_manager.cc:1398] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:55:58.112551 7275 ts_tablet_manager.cc:1398] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:55:58.112855 7257 heartbeater.cc:503] Master 127.5.40.62:37895 was elected leader, sending a full tablet report...
W20240501 19:55:58.248490 6861 tablet.cc:2369] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:55:58.286456 7282 raft_consensus.cc:482] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:55:58.286643 7282 raft_consensus.cc:504] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } }
I20240501 19:55:58.287716 7282 leader_election.cc:288] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 684d010e5b4c428691cd8936e85a77ec (127.5.40.2:40497), e75dbd1c455a4dfd9a041ee3d5cac0c3 (127.5.40.1:46275)
I20240501 19:55:58.291080 6814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f894bc2ae25e4fe9ab52c11f64094b2c" candidate_uuid: "1f04154b01f44df6b969d7abc20a41ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684d010e5b4c428691cd8936e85a77ec" is_pre_election: true
I20240501 19:55:58.291369 6814 raft_consensus.cc:2454] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f04154b01f44df6b969d7abc20a41ed in term 0.
I20240501 19:55:58.291703 6683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f894bc2ae25e4fe9ab52c11f64094b2c" candidate_uuid: "1f04154b01f44df6b969d7abc20a41ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" is_pre_election: true
I20240501 19:55:58.291872 7146 leader_election.cc:302] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f04154b01f44df6b969d7abc20a41ed, 684d010e5b4c428691cd8936e85a77ec; no voters:
I20240501 19:55:58.291955 6683 raft_consensus.cc:2454] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f04154b01f44df6b969d7abc20a41ed in term 0.
I20240501 19:55:58.292109 7282 raft_consensus.cc:2781] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:55:58.292179 7282 raft_consensus.cc:482] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:55:58.292220 7282 raft_consensus.cc:3037] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:58.293123 7282 raft_consensus.cc:504] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } }
I20240501 19:55:58.293452 7282 leader_election.cc:288] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [CANDIDATE]: Term 1 election: Requested vote from peers 684d010e5b4c428691cd8936e85a77ec (127.5.40.2:40497), e75dbd1c455a4dfd9a041ee3d5cac0c3 (127.5.40.1:46275)
I20240501 19:55:58.293831 6814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f894bc2ae25e4fe9ab52c11f64094b2c" candidate_uuid: "1f04154b01f44df6b969d7abc20a41ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684d010e5b4c428691cd8936e85a77ec"
I20240501 19:55:58.293843 6683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f894bc2ae25e4fe9ab52c11f64094b2c" candidate_uuid: "1f04154b01f44df6b969d7abc20a41ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3"
I20240501 19:55:58.293938 6814 raft_consensus.cc:3037] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:58.293973 6683 raft_consensus.cc:3037] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:58.294763 6814 raft_consensus.cc:2454] T f894bc2ae25e4fe9ab52c11f64094b2c P 684d010e5b4c428691cd8936e85a77ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f04154b01f44df6b969d7abc20a41ed in term 1.
I20240501 19:55:58.294799 6683 raft_consensus.cc:2454] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f04154b01f44df6b969d7abc20a41ed in term 1.
I20240501 19:55:58.295097 7146 leader_election.cc:302] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f04154b01f44df6b969d7abc20a41ed, 684d010e5b4c428691cd8936e85a77ec; no voters:
I20240501 19:55:58.295287 7282 raft_consensus.cc:2781] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:58.295539 7282 raft_consensus.cc:686] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [term 1 LEADER]: Becoming Leader. State: Replica: 1f04154b01f44df6b969d7abc20a41ed, State: Running, Role: LEADER
I20240501 19:55:58.295866 7282 consensus_queue.cc:237] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } }
I20240501 19:55:58.298898 6539 catalog_manager.cc:5484] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f04154b01f44df6b969d7abc20a41ed (127.5.40.5). New cstate: current_term: 1 leader_uuid: "1f04154b01f44df6b969d7abc20a41ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f04154b01f44df6b969d7abc20a41ed" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 38413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e75dbd1c455a4dfd9a041ee3d5cac0c3" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 46275 } health_report { overall_health: UNKNOWN } } }
W20240501 19:55:58.301359 7258 tablet.cc:2369] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:55:58.323223 6730 tablet.cc:2369] T f894bc2ae25e4fe9ab52c11f64094b2c P e75dbd1c455a4dfd9a041ee3d5cac0c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:55:58.327860 5280 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:55:58.329299 5280 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e75dbd1c455a4dfd9a041ee3d5cac0c3 to finish bootstrapping
I20240501 19:55:58.334237 5280 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 684d010e5b4c428691cd8936e85a77ec to finish bootstrapping
I20240501 19:55:58.339180 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 61e6abec1ce444cdb5a4b05a094c1300 to finish bootstrapping
I20240501 19:55:58.344713 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f4d410a8669c48bea56dae6c4070858f to finish bootstrapping
I20240501 19:55:58.350157 5280 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1f04154b01f44df6b969d7abc20a41ed to finish bootstrapping
I20240501 19:55:58.355139 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 6601
I20240501 19:55:58.543414 6926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.544658 7059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.545289 7191 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:58.545970 6794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.616264 7191 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:58.616716 7059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.616734 6926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.616948 6794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.683233 7282 consensus_queue.cc:1035] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "684d010e5b4c428691cd8936e85a77ec" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:55:58.687844 7146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.1:46275: connect: Connection refused (error 111)
W20240501 19:55:58.688683 7146 consensus_peers.cc:485] T f894bc2ae25e4fe9ab52c11f64094b2c P 1f04154b01f44df6b969d7abc20a41ed -> Peer e75dbd1c455a4dfd9a041ee3d5cac0c3 (127.5.40.1:46275): Couldn't send request to peer e75dbd1c455a4dfd9a041ee3d5cac0c3. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.1:46275: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:55:58.709446 6993 heartbeater.cc:503] Master 127.5.40.62:37895 was elected leader, sending a full tablet report...
I20240501 19:55:58.712630 7191 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:58.712857 7059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.714021 6794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.721726 6926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.892807 7125 heartbeater.cc:503] Master 127.5.40.62:37895 was elected leader, sending a full tablet report...
I20240501 19:55:58.926910 6926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.928092 7191 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:58.928350 6794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.929245 7059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:58.998849 6926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:59.000380 7191 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:59.003916 7059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:59.010470 6794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:59.112365 6794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:59.112495 7191 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:55:59.113103 6926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:59.114346 7059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:55:59.145951 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 6732
I20240501 19:55:59.150909 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 6863
I20240501 19:55:59.155876 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 6996
I20240501 19:55:59.160270 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 7128
I20240501 19:55:59.164980 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 6510
2024-05-01T19:55:59Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (2380 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2380 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240501 19:55:59.185773 5280 test_util.cc:268] Using random seed: -535050392
I20240501 19:55:59.186511 5280 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:55:59.186553 5280 ts_itest-base.cc:113] --------------
I20240501 19:55:59.186568 5280 ts_itest-base.cc:114] 7 tablet servers
I20240501 19:55:59.186581 5280 ts_itest-base.cc:115] 1 replicas per TS
I20240501 19:55:59.186594 5280 ts_itest-base.cc:116] --------------
2024-05-01T19:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:59Z Disabled control of system clock
I20240501 19:55:59.193953 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.40.62:40123
--webserver_interface=127.5.40.62
--webserver_port=0
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.40.62:40123
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 19:55:59.300158 7478 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 19:55:59.300436 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.300506 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.300576 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.303720 7478 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:59.303817 7478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:59.303857 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.303894 7478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:59.303925 7478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:59.307705 7478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.40.62:40123
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.40.62:40123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.40.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.308657 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.309790 7478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.317307 7486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.317714 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.318189 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.318820 7478 server_base.cc:1034] running on GCE node
I20240501 19:55:59.319384 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.320377 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.321579 7478 hybrid_clock.cc:648] HybridClock initialized: now 1714593359321554 us; error 44 us; skew 500 ppm
I20240501 19:55:59.323707 7478 webserver.cc:460] Webserver started at http://127.5.40.62:34927/ using document root <none> and password file <none>
I20240501 19:55:59.324281 7478 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.324342 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.324524 7478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.326234 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "da61a815e04145ba9603037aebd0daa3"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.326718 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "da61a815e04145ba9603037aebd0daa3"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.330117 7478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 19:55:59.332403 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.333463 7478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:55:59.333581 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
uuid: "da61a815e04145ba9603037aebd0daa3"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.333673 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.346092 7478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.346760 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.346925 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.353355 7478 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.62:40123
I20240501 19:55:59.353374 7544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.62:40123 every 8 connection(s)
I20240501 19:55:59.354327 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:55:59.356909 7545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:59.358512 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 7478
I20240501 19:55:59.358634 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:55:59.363158 7545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3: Bootstrap starting.
I20240501 19:55:59.365566 7545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:59.366302 7545 log.cc:824] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:59.368141 7545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3: No bootstrap required, opened a new log
I20240501 19:55:59.370609 7545 raft_consensus.cc:348] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61a815e04145ba9603037aebd0daa3" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 40123 } }
I20240501 19:55:59.370796 7545 raft_consensus.cc:374] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:59.370837 7545 raft_consensus.cc:729] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da61a815e04145ba9603037aebd0daa3, State: Initialized, Role: FOLLOWER
I20240501 19:55:59.371186 7545 consensus_queue.cc:260] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61a815e04145ba9603037aebd0daa3" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 40123 } }
I20240501 19:55:59.371289 7545 raft_consensus.cc:388] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:59.371345 7545 raft_consensus.cc:482] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:59.371424 7545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:59.372196 7545 raft_consensus.cc:504] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61a815e04145ba9603037aebd0daa3" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 40123 } }
I20240501 19:55:59.372519 7545 leader_election.cc:302] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da61a815e04145ba9603037aebd0daa3; no voters:
I20240501 19:55:59.372781 7545 leader_election.cc:288] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:59.373116 7550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:59.373311 7550 raft_consensus.cc:686] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [term 1 LEADER]: Becoming Leader. State: Replica: da61a815e04145ba9603037aebd0daa3, State: Running, Role: LEADER
I20240501 19:55:59.373615 7550 consensus_queue.cc:237] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61a815e04145ba9603037aebd0daa3" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 40123 } }
I20240501 19:55:59.373873 7545 sys_catalog.cc:564] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:59.375200 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da61a815e04145ba9603037aebd0daa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61a815e04145ba9603037aebd0daa3" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 40123 } } }
I20240501 19:55:59.375361 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:59.375205 7552 sys_catalog.cc:455] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da61a815e04145ba9603037aebd0daa3. Latest consensus state: current_term: 1 leader_uuid: "da61a815e04145ba9603037aebd0daa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61a815e04145ba9603037aebd0daa3" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 40123 } } }
I20240501 19:55:59.375540 7552 sys_catalog.cc:458] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3 [sys.catalog]: This master's current role is: LEADER
W20240501 19:55:59.378712 7565 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:55:59.378782 7565 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:55:59.378876 7560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:59.379837 7560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:59.384821 7560 catalog_manager.cc:1347] Generated new cluster ID: 01764694e7da4485b86f881feff05a1f
I20240501 19:55:59.384925 7560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:59.399833 7560 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:59.400780 7560 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:59.409466 7560 catalog_manager.cc:5857] T 00000000000000000000000000000000 P da61a815e04145ba9603037aebd0daa3: Generated new TSK 0
I20240501 19:55:59.410331 7560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:59.424572 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.1:0
--local_ip_for_outbound_sockets=127.5.40.1
--webserver_interface=127.5.40.1
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:55:59.530349 7569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.530575 7569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.530622 7569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.533748 7569 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:59.533821 7569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.533897 7569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.1
I20240501 19:55:59.537700 7569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.40.1
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.7569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.1
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.538640 7569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.539794 7569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.548215 7574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.548759 7575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.549185 7577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.549790 7569 server_base.cc:1034] running on GCE node
I20240501 19:55:59.550204 7569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.550794 7569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.551950 7569 hybrid_clock.cc:648] HybridClock initialized: now 1714593359551930 us; error 29 us; skew 500 ppm
I20240501 19:55:59.553889 7569 webserver.cc:460] Webserver started at http://127.5.40.1:38943/ using document root <none> and password file <none>
I20240501 19:55:59.554461 7569 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.554520 7569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.554700 7569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.556437 7569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5fc8da5aaa7b4840bca58608368e0466"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.556900 7569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5fc8da5aaa7b4840bca58608368e0466"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.560218 7569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:59.562371 7583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.563385 7569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.563508 7569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5fc8da5aaa7b4840bca58608368e0466"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.563596 7569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.582176 7569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.582880 7569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.583045 7569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.583680 7569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.584595 7569 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.584641 7569 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.584686 7569 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.584713 7569 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.593271 7569 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.1:43301
I20240501 19:55:59.593302 7698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.1:43301 every 8 connection(s)
I20240501 19:55:59.594223 7569 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:55:59.600389 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 7569
I20240501 19:55:59.600513 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:55:59.602712 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.2:0
--local_ip_for_outbound_sockets=127.5.40.2
--webserver_interface=127.5.40.2
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:55:59.602820 7699 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:55:59.603051 7699 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.603721 7699 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:55:59.605345 7505 ts_manager.cc:194] Registered new tserver with Master: 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:55:59.606556 7505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.1:38047
W20240501 19:55:59.709012 7702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.709226 7702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.709271 7702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.712307 7702 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:59.712373 7702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.712455 7702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.2
I20240501 19:55:59.716123 7702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.40.2
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.7702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.2
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.717001 7702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.718093 7702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.726055 7707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.726488 7710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.727087 7708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.727643 7702 server_base.cc:1034] running on GCE node
I20240501 19:55:59.728035 7702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.728597 7702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.729743 7702 hybrid_clock.cc:648] HybridClock initialized: now 1714593359729724 us; error 35 us; skew 500 ppm
I20240501 19:55:59.731614 7702 webserver.cc:460] Webserver started at http://127.5.40.2:42783/ using document root <none> and password file <none>
I20240501 19:55:59.732167 7702 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.732221 7702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.732403 7702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.734042 7702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "09e60f5cde5644f78e51390fce744186"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.734498 7702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "09e60f5cde5644f78e51390fce744186"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.737782 7702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240501 19:55:59.739948 7716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.740954 7702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 19:55:59.741056 7702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "09e60f5cde5644f78e51390fce744186"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.741147 7702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.763424 7702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.764130 7702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.764292 7702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.764851 7702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.765740 7702 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.765786 7702 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.765838 7702 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.765868 7702 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.773900 7702 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.2:39699
I20240501 19:55:59.773962 7831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.2:39699 every 8 connection(s)
I20240501 19:55:59.774811 7702 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:55:59.778205 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 7702
I20240501 19:55:59.778303 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:55:59.780616 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.3:0
--local_ip_for_outbound_sockets=127.5.40.3
--webserver_interface=127.5.40.3
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:55:59.783867 7832 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:55:59.784075 7832 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.784555 7832 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:55:59.785502 7505 ts_manager.cc:194] Registered new tserver with Master: 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:55:59.786108 7505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.2:58077
W20240501 19:55:59.886802 7835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.887025 7835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.887089 7835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.890134 7835 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:59.890208 7835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.890296 7835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.3
I20240501 19:55:59.893846 7835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.40.3
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.7835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.3
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.894731 7835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.895875 7835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.903981 7840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.904558 7843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.905066 7841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.905686 7835 server_base.cc:1034] running on GCE node
I20240501 19:55:59.906021 7835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.906554 7835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.907688 7835 hybrid_clock.cc:648] HybridClock initialized: now 1714593359907670 us; error 33 us; skew 500 ppm
I20240501 19:55:59.909494 7835 webserver.cc:460] Webserver started at http://127.5.40.3:33385/ using document root <none> and password file <none>
I20240501 19:55:59.910064 7835 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.910120 7835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.910303 7835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.911989 7835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.912444 7835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.915607 7835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:59.917661 7849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.918654 7835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.918746 7835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:55:59.918833 7835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.934751 7835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.935364 7835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.935535 7835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.936045 7835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.936939 7835 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.936996 7835 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.937049 7835 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.937076 7835 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.945350 7835 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.3:41917
I20240501 19:55:59.945458 7963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.3:41917 every 8 connection(s)
I20240501 19:55:59.946373 7835 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:55:59.954881 7964 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:55:59.955102 7964 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.955639 7964 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:55:59.956547 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 7835
I20240501 19:55:59.956583 7505 ts_manager.cc:194] Registered new tserver with Master: 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3:41917)
I20240501 19:55:59.956642 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:55:59.957094 7505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.3:36737
I20240501 19:55:59.958417 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.4:0
--local_ip_for_outbound_sockets=127.5.40.4
--webserver_interface=127.5.40.4
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:00.062003 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:00.062238 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:00.062299 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:00.065459 7967 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:00.065539 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:00.065632 7967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.4
I20240501 19:56:00.069588 7967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.40.4
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.7967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.4
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:00.070580 7967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:00.071779 7967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:00.080404 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:00.080899 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:00.081382 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:00.081921 7967 server_base.cc:1034] running on GCE node
I20240501 19:56:00.082348 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:00.082990 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:00.084137 7967 hybrid_clock.cc:648] HybridClock initialized: now 1714593360084132 us; error 29 us; skew 500 ppm
I20240501 19:56:00.086117 7967 webserver.cc:460] Webserver started at http://127.5.40.4:33907/ using document root <none> and password file <none>
I20240501 19:56:00.086710 7967 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:00.086776 7967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:00.086962 7967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:00.088711 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fb4dc8d240984a1c9e05c8a99afe2243"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.089202 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fb4dc8d240984a1c9e05c8a99afe2243"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.092643 7967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:00.095010 7981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.096159 7967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:00.096277 7967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fb4dc8d240984a1c9e05c8a99afe2243"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.096376 7967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:00.111729 7967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:00.112458 7967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:00.112629 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:00.113214 7967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:00.114137 7967 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:00.114197 7967 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.114251 7967 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:00.114280 7967 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.123024 7967 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.4:34385
I20240501 19:56:00.123056 8094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.4:34385 every 8 connection(s)
I20240501 19:56:00.124014 7967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:00.133440 8095 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:00.133621 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 7967
I20240501 19:56:00.133700 8095 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:00.133728 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:56:00.134508 8095 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:00.135596 7505 ts_manager.cc:194] Registered new tserver with Master: fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4:34385)
I20240501 19:56:00.135918 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.5:0
--local_ip_for_outbound_sockets=127.5.40.5
--webserver_interface=127.5.40.5
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:00.136196 7505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.4:38343
W20240501 19:56:00.243932 8098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:00.244217 8098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:00.244290 8098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:00.247653 8098 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:00.247733 8098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:00.247824 8098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.5
I20240501 19:56:00.251641 8098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.40.5
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.8098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.5
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:00.252609 8098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:00.253767 8098 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:00.262332 8107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:00.262676 8103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:00.263231 8104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:00.263938 8098 server_base.cc:1034] running on GCE node
I20240501 19:56:00.264340 8098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:00.264951 8098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:00.266103 8098 hybrid_clock.cc:648] HybridClock initialized: now 1714593360266063 us; error 60 us; skew 500 ppm
I20240501 19:56:00.268147 8098 webserver.cc:460] Webserver started at http://127.5.40.5:35379/ using document root <none> and password file <none>
I20240501 19:56:00.268714 8098 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:00.268775 8098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:00.268961 8098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:00.270606 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "34439dd6bd754595b7219fe0a134e1eb"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.271103 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "34439dd6bd754595b7219fe0a134e1eb"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.274531 8098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:00.277352 8113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.278409 8098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:00.278503 8098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "34439dd6bd754595b7219fe0a134e1eb"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.278582 8098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:00.289278 8098 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:00.289902 8098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:00.290045 8098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:00.290555 8098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:00.291411 8098 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:00.291515 8098 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.291560 8098 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:00.291596 8098 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.299628 8098 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.5:40713
I20240501 19:56:00.299682 8226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.5:40713 every 8 connection(s)
I20240501 19:56:00.300606 8098 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:00.301659 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 8098
I20240501 19:56:00.301759 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:56:00.304083 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.6:0
--local_ip_for_outbound_sockets=127.5.40.6
--webserver_interface=127.5.40.6
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:00.309875 8227 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:00.310093 8227 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:00.310601 8227 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:00.311609 7505 ts_manager.cc:194] Registered new tserver with Master: 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5:40713)
I20240501 19:56:00.312139 7505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.5:54167
W20240501 19:56:00.411252 8230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:00.411494 8230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:00.411556 8230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:00.414574 8230 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:00.414654 8230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:00.414758 8230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.6
I20240501 19:56:00.418602 8230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.40.6
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.8230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.6
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:00.419577 8230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:00.420665 8230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:00.429033 8235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:00.429626 8236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:00.429919 8238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:00.430624 8230 server_base.cc:1034] running on GCE node
I20240501 19:56:00.431010 8230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:00.431620 8230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:00.432780 8230 hybrid_clock.cc:648] HybridClock initialized: now 1714593360432763 us; error 33 us; skew 500 ppm
I20240501 19:56:00.434890 8230 webserver.cc:460] Webserver started at http://127.5.40.6:38621/ using document root <none> and password file <none>
I20240501 19:56:00.435514 8230 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:00.435581 8230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:00.435760 8230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:00.437515 8230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9d482feaacda4c8ab604c21f0c4b718b"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.438016 8230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9d482feaacda4c8ab604c21f0c4b718b"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.441426 8230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:00.443640 8244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.444722 8230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:00.444834 8230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9d482feaacda4c8ab604c21f0c4b718b"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.444919 8230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:00.467262 8230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:00.467994 8230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:00.468150 8230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:00.468681 8230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:00.469543 8230 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:00.469588 8230 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.469640 8230 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:00.469668 8230 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.477914 8230 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.6:32985
I20240501 19:56:00.477944 8358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.6:32985 every 8 connection(s)
I20240501 19:56:00.478873 8230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:00.479794 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 8230
I20240501 19:56:00.479898 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:56:00.482219 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.7:0
--local_ip_for_outbound_sockets=127.5.40.7
--webserver_interface=127.5.40.7
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:00.489161 8359 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:00.489398 8359 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:00.489960 8359 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:00.491024 7505 ts_manager.cc:194] Registered new tserver with Master: 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6:32985)
I20240501 19:56:00.491818 7505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.6:57655
W20240501 19:56:00.591135 8362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:00.591358 8362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:00.591400 8362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:00.594492 8362 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:00.594563 8362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:00.594655 8362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.7
I20240501 19:56:00.598259 8362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.40.7
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.8362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.7
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:00.599123 8362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:00.600320 8362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:00.608634 8367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:00.609015 8368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:00.609308 7699 heartbeater.cc:503] Master 127.5.40.62:40123 was elected leader, sending a full tablet report...
W20240501 19:56:00.609645 8370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:00.610185 8362 server_base.cc:1034] running on GCE node
I20240501 19:56:00.610674 8362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:00.611428 8362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:00.612633 8362 hybrid_clock.cc:648] HybridClock initialized: now 1714593360612616 us; error 32 us; skew 500 ppm
I20240501 19:56:00.614581 8362 webserver.cc:460] Webserver started at http://127.5.40.7:37059/ using document root <none> and password file <none>
I20240501 19:56:00.615145 8362 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:00.615202 8362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:00.615387 8362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:00.617082 8362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.617525 8362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.620731 8362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:00.622952 8376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.624024 8362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:00.624126 8362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:00.624213 8362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:00.643538 8362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:00.644204 8362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:00.644364 8362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:00.644924 8362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:00.645809 8362 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:00.645866 8362 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.645920 8362 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:00.645948 8362 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.654302 8362 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.7:46465
I20240501 19:56:00.654383 8490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.7:46465 every 8 connection(s)
I20240501 19:56:00.655236 8362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:00.658062 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 8362
I20240501 19:56:00.658175 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:56:00.663822 8491 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:00.664017 8491 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:00.664484 8491 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:00.665374 7505 ts_manager.cc:194] Registered new tserver with Master: b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7:46465)
I20240501 19:56:00.665886 7505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.7:39515
I20240501 19:56:00.672188 5280 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 19:56:00.680949 7505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45670:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:56:00.696307 8025 tablet_service.cc:1467] Processing CreateTablet for tablet a84bdf4685da48f18a48e8f87355fc3f (DEFAULT_TABLE table=TestTable [id=822ffe74ed8e47aca3589df3bac7f03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:00.697158 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84bdf4685da48f18a48e8f87355fc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.702051 8505 tablet_bootstrap.cc:492] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Bootstrap starting.
I20240501 19:56:00.703989 8505 tablet_bootstrap.cc:654] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.704617 8505 log.cc:824] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:00.706099 8505 tablet_bootstrap.cc:492] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: No bootstrap required, opened a new log
I20240501 19:56:00.706265 8505 ts_tablet_manager.cc:1367] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:00.708752 8505 raft_consensus.cc:348] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } }
I20240501 19:56:00.708923 8505 raft_consensus.cc:374] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.708956 8505 raft_consensus.cc:729] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4dc8d240984a1c9e05c8a99afe2243, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.709298 8505 consensus_queue.cc:260] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } }
I20240501 19:56:00.709401 8505 raft_consensus.cc:388] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:00.709445 8505 raft_consensus.cc:482] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:00.709527 8505 raft_consensus.cc:3037] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.710583 8505 raft_consensus.cc:504] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } }
I20240501 19:56:00.710866 8505 leader_election.cc:302] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4dc8d240984a1c9e05c8a99afe2243; no voters:
I20240501 19:56:00.711107 8505 leader_election.cc:288] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:00.711210 8507 raft_consensus.cc:2781] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:00.711424 8507 raft_consensus.cc:686] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 LEADER]: Becoming Leader. State: Replica: fb4dc8d240984a1c9e05c8a99afe2243, State: Running, Role: LEADER
I20240501 19:56:00.711743 8095 heartbeater.cc:503] Master 127.5.40.62:40123 was elected leader, sending a full tablet report...
I20240501 19:56:00.711966 8505 ts_tablet_manager.cc:1398] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 19:56:00.711990 8507 consensus_queue.cc:237] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } }
I20240501 19:56:00.715516 7505 catalog_manager.cc:5484] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:00.722373 5280 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 19:56:00.723623 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5fc8da5aaa7b4840bca58608368e0466 to finish bootstrapping
I20240501 19:56:00.729524 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 09e60f5cde5644f78e51390fce744186 to finish bootstrapping
I20240501 19:56:00.735165 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 29cb7a4d71c44e58956f1ea5afa2e2c4 to finish bootstrapping
I20240501 19:56:00.740991 5280 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fb4dc8d240984a1c9e05c8a99afe2243 to finish bootstrapping
I20240501 19:56:00.745771 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 34439dd6bd754595b7219fe0a134e1eb to finish bootstrapping
I20240501 19:56:00.751559 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9d482feaacda4c8ab604c21f0c4b718b to finish bootstrapping
I20240501 19:56:00.757022 5280 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b39e25d5bdc047c3ac97e42fa43d2aa5 to finish bootstrapping
I20240501 19:56:00.762498 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 7835
I20240501 19:56:00.767041 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 7967
I20240501 19:56:00.771476 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 8098
I20240501 19:56:00.776854 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 8230
I20240501 19:56:00.782441 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 8362
I20240501 19:56:00.788007 7832 heartbeater.cc:503] Master 127.5.40.62:40123 was elected leader, sending a full tablet report...
I20240501 19:56:04.538087 7505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45670:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:04.550254 7765 tablet_service.cc:1467] Processing CreateTablet for tablet 6f10d321e5494c9b933877a1c74ca36d (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ca20120f9b74a50b2a14f29cc088bf2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.550254 7631 tablet_service.cc:1467] Processing CreateTablet for tablet 0fead6e4699349b4a57548986c444c90 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ca20120f9b74a50b2a14f29cc088bf2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.550254 7630 tablet_service.cc:1467] Processing CreateTablet for tablet 2458bf9cc6ae4ce181f9073df78ee70d (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ca20120f9b74a50b2a14f29cc088bf2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.551147 7765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f10d321e5494c9b933877a1c74ca36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.551192 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fead6e4699349b4a57548986c444c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.551509 7630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2458bf9cc6ae4ce181f9073df78ee70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.556300 8527 tablet_bootstrap.cc:492] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186: Bootstrap starting.
I20240501 19:56:04.556301 8528 tablet_bootstrap.cc:492] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466: Bootstrap starting.
I20240501 19:56:04.558310 8527 tablet_bootstrap.cc:654] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.558574 8528 tablet_bootstrap.cc:654] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.559007 8527 log.cc:824] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:04.559146 8528 log.cc:824] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:04.560503 8528 tablet_bootstrap.cc:492] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466: No bootstrap required, opened a new log
I20240501 19:56:04.560508 8527 tablet_bootstrap.cc:492] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186: No bootstrap required, opened a new log
I20240501 19:56:04.560655 8527 ts_tablet_manager.cc:1367] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:04.560657 8528 ts_tablet_manager.cc:1367] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 19:56:04.563624 8528 raft_consensus.cc:348] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.563624 8527 raft_consensus.cc:348] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.563807 8528 raft_consensus.cc:374] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.563807 8527 raft_consensus.cc:374] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.563850 8528 raft_consensus.cc:729] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.563863 8527 raft_consensus.cc:729] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.564258 8528 consensus_queue.cc:260] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.564258 8527 consensus_queue.cc:260] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.564378 8528 raft_consensus.cc:388] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:04.564378 8527 raft_consensus.cc:388] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:04.564441 8527 raft_consensus.cc:482] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:04.564448 8528 raft_consensus.cc:482] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:04.564544 8527 raft_consensus.cc:3037] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.564543 8528 raft_consensus.cc:3037] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.565295 8527 raft_consensus.cc:504] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.565310 8528 raft_consensus.cc:504] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.565610 8527 leader_election.cc:302] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186; no voters:
I20240501 19:56:04.565610 8528 leader_election.cc:302] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:04.565891 8528 leader_election.cc:288] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:04.565899 8527 leader_election.cc:288] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:04.565997 8531 raft_consensus.cc:2781] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.566021 8532 raft_consensus.cc:2781] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.566246 8531 raft_consensus.cc:686] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:04.566280 8532 raft_consensus.cc:686] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Leader. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:04.566593 8531 consensus_queue.cc:237] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.566658 8532 consensus_queue.cc:237] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.566787 8528 ts_tablet_manager.cc:1398] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20240501 19:56:04.566785 8527 ts_tablet_manager.cc:1398] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 19:56:04.567492 8528 tablet_bootstrap.cc:492] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: Bootstrap starting.
I20240501 19:56:04.569715 7505 catalog_manager.cc:5484] T 0fead6e4699349b4a57548986c444c90 P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1). New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:04.570227 7509 catalog_manager.cc:5484] T 6f10d321e5494c9b933877a1c74ca36d P 09e60f5cde5644f78e51390fce744186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09e60f5cde5644f78e51390fce744186 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:04.571014 8528 tablet_bootstrap.cc:654] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.572525 8528 tablet_bootstrap.cc:492] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: No bootstrap required, opened a new log
I20240501 19:56:04.572607 8528 ts_tablet_manager.cc:1367] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:04.573135 8528 raft_consensus.cc:348] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.573227 8528 raft_consensus.cc:374] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.573256 8528 raft_consensus.cc:729] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.573379 8528 consensus_queue.cc:260] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.573482 8528 raft_consensus.cc:388] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:04.573513 8528 raft_consensus.cc:482] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:04.573559 8528 raft_consensus.cc:3037] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.574362 8528 raft_consensus.cc:504] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.574486 8528 leader_election.cc:302] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:04.574579 8528 leader_election.cc:288] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:04.574647 8534 raft_consensus.cc:2781] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.574785 8528 ts_tablet_manager.cc:1398] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:04.574784 8534 raft_consensus.cc:686] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:04.574995 8534 consensus_queue.cc:237] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.576336 7509 catalog_manager.cc:5484] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1). New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:04.690202 8544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.690481 8544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.693600 8544 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 69.8265 ms
time per row: 0.0349132 ms
I20240501 19:56:04.802848 7500 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45670:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:04.808095 7765 tablet_service.cc:1467] Processing CreateTablet for tablet eb4733c007f34c4a850f9ac32a778b91 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a78b96eee0024a568693d7827bc012c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.808095 7764 tablet_service.cc:1467] Processing CreateTablet for tablet 410cc1fb888a4d2f945c7518ee2b9580 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a78b96eee0024a568693d7827bc012c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.808274 7631 tablet_service.cc:1467] Processing CreateTablet for tablet e4846083047b478a977ae8e85e0f958d (DEFAULT_TABLE table=rebalance_test_table_1 [id=a78b96eee0024a568693d7827bc012c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.808455 7765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4733c007f34c4a850f9ac32a778b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.808547 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4846083047b478a977ae8e85e0f958d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.808626 7764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410cc1fb888a4d2f945c7518ee2b9580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.810062 8527 tablet_bootstrap.cc:492] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: Bootstrap starting.
I20240501 19:56:04.810478 8528 tablet_bootstrap.cc:492] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: Bootstrap starting.
I20240501 19:56:04.810931 8527 tablet_bootstrap.cc:654] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.811235 8528 tablet_bootstrap.cc:654] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.812026 8527 tablet_bootstrap.cc:492] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: No bootstrap required, opened a new log
I20240501 19:56:04.812106 8527 ts_tablet_manager.cc:1367] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.812179 8528 tablet_bootstrap.cc:492] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: No bootstrap required, opened a new log
I20240501 19:56:04.812260 8528 ts_tablet_manager.cc:1367] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.812626 8528 raft_consensus.cc:348] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.812649 8527 raft_consensus.cc:348] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.812736 8527 raft_consensus.cc:374] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.812736 8528 raft_consensus.cc:374] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.812764 8527 raft_consensus.cc:729] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.812770 8528 raft_consensus.cc:729] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.812891 8527 consensus_queue.cc:260] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.812892 8528 consensus_queue.cc:260] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.812958 8527 raft_consensus.cc:388] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:04.812985 8528 raft_consensus.cc:388] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:04.812987 8527 raft_consensus.cc:482] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:04.813012 8528 raft_consensus.cc:482] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:04.813024 8527 raft_consensus.cc:3037] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.813040 8528 raft_consensus.cc:3037] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.813714 8528 raft_consensus.cc:504] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.813724 8527 raft_consensus.cc:504] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.813812 8528 leader_election.cc:302] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:04.813822 8527 leader_election.cc:302] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186; no voters:
I20240501 19:56:04.813880 8528 leader_election.cc:288] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:04.813903 8527 leader_election.cc:288] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:04.813973 8534 raft_consensus.cc:2781] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.813987 8536 raft_consensus.cc:2781] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.814076 8534 raft_consensus.cc:686] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:04.814182 8534 consensus_queue.cc:237] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:04.814427 8536 raft_consensus.cc:686] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Leader. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:04.814462 8527 ts_tablet_manager.cc:1398] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:04.814611 8527 tablet_bootstrap.cc:492] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186: Bootstrap starting.
I20240501 19:56:04.814563 8536 consensus_queue.cc:237] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.815505 8528 ts_tablet_manager.cc:1398] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:04.815543 8527 tablet_bootstrap.cc:654] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.815953 7500 catalog_manager.cc:5484] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1). New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:04.816102 7509 catalog_manager.cc:5484] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09e60f5cde5644f78e51390fce744186 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:04.818217 8527 tablet_bootstrap.cc:492] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186: No bootstrap required, opened a new log
I20240501 19:56:04.818302 8527 ts_tablet_manager.cc:1367] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:04.818710 8527 raft_consensus.cc:348] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.818790 8527 raft_consensus.cc:374] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.818817 8527 raft_consensus.cc:729] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.818935 8527 consensus_queue.cc:260] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.819005 8527 raft_consensus.cc:388] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:04.819025 8527 raft_consensus.cc:482] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:04.819052 8527 raft_consensus.cc:3037] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.819913 8527 raft_consensus.cc:504] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.820045 8527 leader_election.cc:302] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186; no voters:
I20240501 19:56:04.820142 8527 leader_election.cc:288] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:04.820204 8536 raft_consensus.cc:2781] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.820304 8536 raft_consensus.cc:686] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Leader. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:04.820341 8527 ts_tablet_manager.cc:1398] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:04.820411 8536 consensus_queue.cc:237] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:04.821923 7509 catalog_manager.cc:5484] T 410cc1fb888a4d2f945c7518ee2b9580 P 09e60f5cde5644f78e51390fce744186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09e60f5cde5644f78e51390fce744186 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:04.930744 8586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.931012 8586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.934022 8586 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 62.2208 ms
time per row: 0.0311104 ms
I20240501 19:56:05.035154 7501 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45670:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:05.040696 7764 tablet_service.cc:1467] Processing CreateTablet for tablet 408efa4d06cb40938b03d6d988a99302 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e19fd899e1cb46f19ec797cf6ef227e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.040688 7631 tablet_service.cc:1467] Processing CreateTablet for tablet 9b77e626fdd047dab6bbb0a271ada708 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e19fd899e1cb46f19ec797cf6ef227e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.040772 7630 tablet_service.cc:1467] Processing CreateTablet for tablet d3f595797050475f8e8003c3cdf6d96a (DEFAULT_TABLE table=rebalance_test_table_2 [id=e19fd899e1cb46f19ec797cf6ef227e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.041006 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b77e626fdd047dab6bbb0a271ada708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.041006 7764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 408efa4d06cb40938b03d6d988a99302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.041163 7630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f595797050475f8e8003c3cdf6d96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.042768 8528 tablet_bootstrap.cc:492] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: Bootstrap starting.
I20240501 19:56:05.043038 8527 tablet_bootstrap.cc:492] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: Bootstrap starting.
I20240501 19:56:05.043608 8528 tablet_bootstrap.cc:654] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.043728 8527 tablet_bootstrap.cc:654] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.044687 8528 tablet_bootstrap.cc:492] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: No bootstrap required, opened a new log
I20240501 19:56:05.044713 8527 tablet_bootstrap.cc:492] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: No bootstrap required, opened a new log
I20240501 19:56:05.044773 8527 ts_tablet_manager.cc:1367] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.044770 8528 ts_tablet_manager.cc:1367] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.045202 8527 raft_consensus.cc:348] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.045223 8528 raft_consensus.cc:348] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.045298 8528 raft_consensus.cc:374] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.045298 8527 raft_consensus.cc:374] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.045327 8528 raft_consensus.cc:729] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.045338 8527 raft_consensus.cc:729] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.045444 8527 consensus_queue.cc:260] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.045444 8528 consensus_queue.cc:260] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.045514 8528 raft_consensus.cc:388] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:05.045514 8527 raft_consensus.cc:388] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:05.045545 8527 raft_consensus.cc:482] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:05.045545 8528 raft_consensus.cc:482] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:05.045586 8528 raft_consensus.cc:3037] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.045588 8527 raft_consensus.cc:3037] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.046360 8528 raft_consensus.cc:504] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.046360 8527 raft_consensus.cc:504] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.046478 8528 leader_election.cc:302] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:05.046478 8527 leader_election.cc:302] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186; no voters:
I20240501 19:56:05.046567 8527 leader_election.cc:288] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:05.046567 8528 leader_election.cc:288] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:05.046634 8534 raft_consensus.cc:2781] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.046634 8536 raft_consensus.cc:2781] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.046732 8536 raft_consensus.cc:686] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Leader. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:05.046774 8527 ts_tablet_manager.cc:1398] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 19:56:05.046854 8536 consensus_queue.cc:237] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.047044 8534 raft_consensus.cc:686] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:05.047073 8528 ts_tablet_manager.cc:1398] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.047168 8534 consensus_queue.cc:237] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.047235 8528 tablet_bootstrap.cc:492] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: Bootstrap starting.
I20240501 19:56:05.048071 8528 tablet_bootstrap.cc:654] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.048578 7501 catalog_manager.cc:5484] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1). New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:05.048609 7505 catalog_manager.cc:5484] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09e60f5cde5644f78e51390fce744186 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:05.049571 8528 tablet_bootstrap.cc:492] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: No bootstrap required, opened a new log
I20240501 19:56:05.049630 8528 ts_tablet_manager.cc:1367] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.049978 8528 raft_consensus.cc:348] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.050047 8528 raft_consensus.cc:374] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.050066 8528 raft_consensus.cc:729] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.050174 8528 consensus_queue.cc:260] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.050246 8528 raft_consensus.cc:388] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:05.050276 8528 raft_consensus.cc:482] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:05.050310 8528 raft_consensus.cc:3037] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.051045 8528 raft_consensus.cc:504] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.051149 8528 leader_election.cc:302] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:05.051229 8528 leader_election.cc:288] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:05.051275 8531 raft_consensus.cc:2781] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.051340 8531 raft_consensus.cc:686] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:05.051460 8528 ts_tablet_manager.cc:1398] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.051465 8531 consensus_queue.cc:237] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.053246 7501 catalog_manager.cc:5484] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1). New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:05.161957 8619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.162205 8619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.165208 8619 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 64.6399 ms
time per row: 0.0323199 ms
I20240501 19:56:05.272578 7500 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45670:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:05.278398 7764 tablet_service.cc:1467] Processing CreateTablet for tablet 75f7f2b5acf54c43b00fec9c033aadc9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=deba21d5b67a4c8a9b2322eca8b8faf1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.278445 7765 tablet_service.cc:1467] Processing CreateTablet for tablet 3b567cf2477443bbb17f0c7701fa1411 (DEFAULT_TABLE table=rebalance_test_table_3 [id=deba21d5b67a4c8a9b2322eca8b8faf1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.278550 7631 tablet_service.cc:1467] Processing CreateTablet for tablet 161af226948248898c90ad36db3f0ec9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=deba21d5b67a4c8a9b2322eca8b8faf1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.278723 7764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f7f2b5acf54c43b00fec9c033aadc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.278808 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161af226948248898c90ad36db3f0ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.278820 7765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b567cf2477443bbb17f0c7701fa1411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.280313 8527 tablet_bootstrap.cc:492] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: Bootstrap starting.
I20240501 19:56:05.280702 8528 tablet_bootstrap.cc:492] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: Bootstrap starting.
I20240501 19:56:05.281157 8527 tablet_bootstrap.cc:654] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.281327 8528 tablet_bootstrap.cc:654] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.282192 8527 tablet_bootstrap.cc:492] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: No bootstrap required, opened a new log
I20240501 19:56:05.282253 8528 tablet_bootstrap.cc:492] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: No bootstrap required, opened a new log
I20240501 19:56:05.282265 8527 ts_tablet_manager.cc:1367] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.282325 8528 ts_tablet_manager.cc:1367] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.282622 8527 raft_consensus.cc:348] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.282704 8527 raft_consensus.cc:374] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.282724 8527 raft_consensus.cc:729] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.282827 8527 consensus_queue.cc:260] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.282902 8527 raft_consensus.cc:388] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:05.282929 8527 raft_consensus.cc:482] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:05.282974 8527 raft_consensus.cc:3037] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.283151 8528 raft_consensus.cc:348] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.283231 8528 raft_consensus.cc:374] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.283259 8528 raft_consensus.cc:729] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.283373 8528 consensus_queue.cc:260] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.283469 8528 raft_consensus.cc:388] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:05.283509 8528 raft_consensus.cc:482] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:05.283555 8528 raft_consensus.cc:3037] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.283721 8527 raft_consensus.cc:504] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.283855 8527 leader_election.cc:302] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186; no voters:
I20240501 19:56:05.283958 8527 leader_election.cc:288] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:05.284008 8532 raft_consensus.cc:2781] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.284098 8532 raft_consensus.cc:686] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Leader. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:05.284204 8532 consensus_queue.cc:237] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.284267 8527 ts_tablet_manager.cc:1398] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.284276 8528 raft_consensus.cc:504] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.284391 8527 tablet_bootstrap.cc:492] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: Bootstrap starting.
I20240501 19:56:05.284373 8528 leader_election.cc:302] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:05.284456 8528 leader_election.cc:288] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:05.284543 8534 raft_consensus.cc:2781] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.284627 8534 raft_consensus.cc:686] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:05.284732 8534 consensus_queue.cc:237] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.284924 8528 ts_tablet_manager.cc:1398] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:05.285231 8527 tablet_bootstrap.cc:654] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.285775 7500 catalog_manager.cc:5484] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09e60f5cde5644f78e51390fce744186 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:05.286303 8527 tablet_bootstrap.cc:492] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: No bootstrap required, opened a new log
I20240501 19:56:05.286391 8527 ts_tablet_manager.cc:1367] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.286537 7501 catalog_manager.cc:5484] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1). New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:05.286810 8527 raft_consensus.cc:348] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.286886 8527 raft_consensus.cc:374] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.286914 8527 raft_consensus.cc:729] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.287011 8527 consensus_queue.cc:260] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.287093 8527 raft_consensus.cc:388] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:05.287134 8527 raft_consensus.cc:482] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:05.287180 8527 raft_consensus.cc:3037] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.287828 8527 raft_consensus.cc:504] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.287925 8527 leader_election.cc:302] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186; no voters:
I20240501 19:56:05.288008 8527 leader_election.cc:288] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:05.288136 8533 raft_consensus.cc:2781] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.288175 8527 ts_tablet_manager.cc:1398] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.288223 8533 raft_consensus.cc:686] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Leader. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:05.288331 8533 consensus_queue.cc:237] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.290815 7501 catalog_manager.cc:5484] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09e60f5cde5644f78e51390fce744186 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:05.403035 8651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.403301 8651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.406392 8651 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 69.9483 ms
time per row: 0.0349741 ms
I20240501 19:56:05.515297 7506 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45670:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:05.520659 7631 tablet_service.cc:1467] Processing CreateTablet for tablet ba539430c50348fc8610a2172966cb08 (DEFAULT_TABLE table=rebalance_test_table_4 [id=886f5e1b079149088d32570e8e24019e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.520829 7630 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5b8c745ecf48a083e700b077ce10ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=886f5e1b079149088d32570e8e24019e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.520973 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba539430c50348fc8610a2172966cb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.520927 7765 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b (DEFAULT_TABLE table=rebalance_test_table_4 [id=886f5e1b079149088d32570e8e24019e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.521090 7630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5b8c745ecf48a083e700b077ce10ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.521183 7765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.522851 8528 tablet_bootstrap.cc:492] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: Bootstrap starting.
I20240501 19:56:05.523300 8527 tablet_bootstrap.cc:492] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: Bootstrap starting.
I20240501 19:56:05.523852 8528 tablet_bootstrap.cc:654] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.524140 8527 tablet_bootstrap.cc:654] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.524914 8528 tablet_bootstrap.cc:492] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: No bootstrap required, opened a new log
I20240501 19:56:05.524999 8528 ts_tablet_manager.cc:1367] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.525133 8527 tablet_bootstrap.cc:492] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: No bootstrap required, opened a new log
I20240501 19:56:05.525207 8527 ts_tablet_manager.cc:1367] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.525408 8528 raft_consensus.cc:348] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.525494 8528 raft_consensus.cc:374] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.525522 8528 raft_consensus.cc:729] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.525555 8527 raft_consensus.cc:348] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.525648 8527 raft_consensus.cc:374] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.525678 8527 raft_consensus.cc:729] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.525651 8528 consensus_queue.cc:260] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.525725 8528 raft_consensus.cc:388] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:05.525763 8528 raft_consensus.cc:482] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:05.525799 8528 raft_consensus.cc:3037] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.525794 8527 consensus_queue.cc:260] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.525867 8527 raft_consensus.cc:388] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:05.525888 8527 raft_consensus.cc:482] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:05.525920 8527 raft_consensus.cc:3037] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.526553 8528 raft_consensus.cc:504] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.526674 8528 leader_election.cc:302] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:05.526691 8527 raft_consensus.cc:504] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.526747 8528 leader_election.cc:288] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:05.526794 8527 leader_election.cc:302] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186; no voters:
I20240501 19:56:05.526813 8534 raft_consensus.cc:2781] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.526885 8534 raft_consensus.cc:686] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:05.526922 8532 raft_consensus.cc:2781] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.526980 8532 raft_consensus.cc:686] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Leader. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:05.526993 8534 consensus_queue.cc:237] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.527077 8532 consensus_queue.cc:237] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } }
I20240501 19:56:05.527143 8528 ts_tablet_manager.cc:1398] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.527292 8528 tablet_bootstrap.cc:492] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466: Bootstrap starting.
I20240501 19:56:05.528177 8528 tablet_bootstrap.cc:654] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.528419 8527 leader_election.cc:288] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:05.528714 8527 ts_tablet_manager.cc:1398] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:05.528762 7506 catalog_manager.cc:5484] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1). New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:05.529188 7501 catalog_manager.cc:5484] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09e60f5cde5644f78e51390fce744186 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:05.530058 8528 tablet_bootstrap.cc:492] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466: No bootstrap required, opened a new log
I20240501 19:56:05.530180 8528 ts_tablet_manager.cc:1367] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:05.530556 8528 raft_consensus.cc:348] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.531158 8528 raft_consensus.cc:374] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.531280 8528 raft_consensus.cc:729] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.531416 8528 consensus_queue.cc:260] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.531548 8528 raft_consensus.cc:388] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:05.531581 8528 raft_consensus.cc:482] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:05.531626 8528 raft_consensus.cc:3037] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.532496 8528 raft_consensus.cc:504] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.532624 8528 leader_election.cc:302] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:05.532737 8528 leader_election.cc:288] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:05.532819 8534 raft_consensus.cc:2781] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.532895 8534 raft_consensus.cc:686] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:05.532981 8528 ts_tablet_manager.cc:1398] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:05.533005 8534 consensus_queue.cc:237] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } }
I20240501 19:56:05.534480 7501 catalog_manager.cc:5484] T ba539430c50348fc8610a2172966cb08 P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1). New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:05.646596 8682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.646849 8682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.649883 8682 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 63.4424 ms
time per row: 0.0317212 ms
I20240501 19:56:05.753657 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.3:41917
--local_ip_for_outbound_sockets=127.5.40.3
--tserver_master_addrs=127.5.40.62:40123
--webserver_port=33385
--webserver_interface=127.5.40.3
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:05.860384 8709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.860608 8709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:05.860654 8709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.863768 8709 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:05.863839 8709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:05.863926 8709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.3
I20240501 19:56:05.867672 8709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.3:41917
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.40.3
--webserver_port=33385
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.8709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.3
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:05.868625 8709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:05.869781 8709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:05.878520 8715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:05.879005 8714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:05.879498 8717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:05.880015 8709 server_base.cc:1034] running on GCE node
I20240501 19:56:05.880424 8709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:05.881019 8709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:05.882179 8709 hybrid_clock.cc:648] HybridClock initialized: now 1714593365882160 us; error 37 us; skew 500 ppm
I20240501 19:56:05.884119 8709 webserver.cc:460] Webserver started at http://127.5.40.3:33385/ using document root <none> and password file <none>
I20240501 19:56:05.884683 8709 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:05.884760 8709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:05.888329 8709 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:05.890259 8723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:05.891271 8709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.891379 8709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-bz40"
I20240501 19:56:05.891839 8709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:05.925282 8709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:05.926115 8709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:05.926278 8709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:05.926841 8709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:05.927762 8709 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:05.927810 8709 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:05.927873 8709 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:05.927897 8709 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:05.935907 8709 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.3:41917
I20240501 19:56:05.935935 8837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.3:41917 every 8 connection(s)
I20240501 19:56:05.936959 8709 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:56:05.938907 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 8709
I20240501 19:56:05.939256 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.4:34385
--local_ip_for_outbound_sockets=127.5.40.4
--tserver_master_addrs=127.5.40.62:40123
--webserver_port=33907
--webserver_interface=127.5.40.4
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:05.945574 8838 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:05.945823 8838 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:05.946480 8838 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:05.947657 7506 ts_manager.cc:194] Re-registered known tserver with Master: 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3:41917)
I20240501 19:56:05.948215 7506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.3:48463
W20240501 19:56:06.047933 8841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:06.048172 8841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:06.048228 8841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:06.051326 8841 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:06.051414 8841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:06.051537 8841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.4
I20240501 19:56:06.055362 8841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.4:34385
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.40.4
--webserver_port=33907
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.8841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.4
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:06.056355 8841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:06.057579 8841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:06.065698 8847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.066241 8848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.066680 8850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:06.067423 8841 server_base.cc:1034] running on GCE node
I20240501 19:56:06.067848 8841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:06.068456 8841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:06.069604 8841 hybrid_clock.cc:648] HybridClock initialized: now 1714593366069589 us; error 33 us; skew 500 ppm
I20240501 19:56:06.071538 8841 webserver.cc:460] Webserver started at http://127.5.40.4:33907/ using document root <none> and password file <none>
I20240501 19:56:06.072082 8841 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:06.072153 8841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:06.075860 8841 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 19:56:06.077957 8856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.078960 8841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:06.079058 8841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fb4dc8d240984a1c9e05c8a99afe2243"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:06.079478 8841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:06.091925 8841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:06.092561 8841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:06.092697 8841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:06.093191 8841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:06.094259 8863 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 19:56:06.097412 8841 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 19:56:06.097537 8841 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 19:56:06.097632 8841 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 19:56:06.099176 8841 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 19:56:06.099226 8841 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 19:56:06.099567 8863 tablet_bootstrap.cc:492] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Bootstrap starting.
I20240501 19:56:06.108660 8841 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.4:34385
I20240501 19:56:06.108700 8971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.4:34385 every 8 connection(s)
I20240501 19:56:06.109671 8841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:06.113699 8863 log.cc:824] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:06.115001 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 8841
I20240501 19:56:06.115420 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.5:40713
--local_ip_for_outbound_sockets=127.5.40.5
--tserver_master_addrs=127.5.40.62:40123
--webserver_port=35379
--webserver_interface=127.5.40.5
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:06.116055 8863 tablet_bootstrap.cc:492] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:06.116492 8863 tablet_bootstrap.cc:492] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Bootstrap complete.
I20240501 19:56:06.117493 8863 ts_tablet_manager.cc:1367] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20240501 19:56:06.118737 8972 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:06.118947 8972 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:06.119493 8972 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:06.120486 8863 raft_consensus.cc:348] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } }
I20240501 19:56:06.120891 8863 raft_consensus.cc:729] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4dc8d240984a1c9e05c8a99afe2243, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.121065 7506 ts_manager.cc:194] Re-registered known tserver with Master: fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4:34385)
I20240501 19:56:06.121387 8863 consensus_queue.cc:260] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } }
I20240501 19:56:06.121554 8863 raft_consensus.cc:388] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:06.121613 8863 raft_consensus.cc:482] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:06.121708 8863 raft_consensus.cc:3037] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:06.121853 7506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.4:49831
I20240501 19:56:06.123788 8863 raft_consensus.cc:504] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } }
I20240501 19:56:06.124158 8863 leader_election.cc:302] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4dc8d240984a1c9e05c8a99afe2243; no voters:
I20240501 19:56:06.124476 8863 leader_election.cc:288] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 19:56:06.124814 8978 raft_consensus.cc:2781] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:06.125056 8978 raft_consensus.cc:686] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [term 2 LEADER]: Becoming Leader. State: Replica: fb4dc8d240984a1c9e05c8a99afe2243, State: Running, Role: LEADER
I20240501 19:56:06.125161 8972 heartbeater.cc:503] Master 127.5.40.62:40123 was elected leader, sending a full tablet report...
I20240501 19:56:06.125355 8863 ts_tablet_manager.cc:1398] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240501 19:56:06.125401 8978 consensus_queue.cc:237] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } }
I20240501 19:56:06.127781 7506 catalog_manager.cc:5484] T a84bdf4685da48f18a48e8f87355fc3f P fb4dc8d240984a1c9e05c8a99afe2243 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:06.225513 8976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:06.225749 8976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:06.225809 8976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:06.229017 8976 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:06.229092 8976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:06.229180 8976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.5
I20240501 19:56:06.232851 8976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.5:40713
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.40.5
--webserver_port=35379
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.8976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.5
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:06.233776 8976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:06.234979 8976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:06.243716 8990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.243896 8992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.244524 8989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:06.245077 8976 server_base.cc:1034] running on GCE node
I20240501 19:56:06.245463 8976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:06.246052 8976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:06.247215 8976 hybrid_clock.cc:648] HybridClock initialized: now 1714593366247198 us; error 31 us; skew 500 ppm
I20240501 19:56:06.249140 8976 webserver.cc:460] Webserver started at http://127.5.40.5:35379/ using document root <none> and password file <none>
I20240501 19:56:06.249691 8976 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:06.249761 8976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:06.253168 8976 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:06.255017 8998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.256024 8976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:06.256126 8976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "34439dd6bd754595b7219fe0a134e1eb"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:06.256531 8976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:06.284024 8976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:06.284664 8976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:06.284821 8976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:06.285377 8976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:06.286300 8976 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:06.286355 8976 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.286408 8976 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:06.286433 8976 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.294198 8976 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.5:40713
I20240501 19:56:06.294229 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.5:40713 every 8 connection(s)
I20240501 19:56:06.295193 8976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:06.303723 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 8976
I20240501 19:56:06.303755 9112 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:06.303984 9112 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:06.304157 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.6:32985
--local_ip_for_outbound_sockets=127.5.40.6
--tserver_master_addrs=127.5.40.62:40123
--webserver_port=38621
--webserver_interface=127.5.40.6
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:06.304596 9112 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:06.305716 7506 ts_manager.cc:194] Re-registered known tserver with Master: 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5:40713)
I20240501 19:56:06.306439 7506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.5:40599
W20240501 19:56:06.411257 9115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:06.411557 9115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:06.411620 9115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:06.414656 9115 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:06.414737 9115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:06.414831 9115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.6
I20240501 19:56:06.418535 9115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.6:32985
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.40.6
--webserver_port=38621
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.9115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.6
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:06.419485 9115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:06.420622 9115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:06.429054 9121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.429307 9122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.429773 9124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:06.430328 9115 server_base.cc:1034] running on GCE node
I20240501 19:56:06.430708 9115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:06.431232 9115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:06.432389 9115 hybrid_clock.cc:648] HybridClock initialized: now 1714593366432373 us; error 28 us; skew 500 ppm
I20240501 19:56:06.434265 9115 webserver.cc:460] Webserver started at http://127.5.40.6:38621/ using document root <none> and password file <none>
I20240501 19:56:06.434803 9115 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:06.434870 9115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:06.438303 9115 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:06.440136 9130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.441149 9115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:06.441244 9115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9d482feaacda4c8ab604c21f0c4b718b"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:06.441686 9115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:06.463955 9115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:06.464617 9115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:06.464769 9115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:06.465319 9115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:06.466197 9115 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:06.466243 9115 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.466284 9115 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:06.466311 9115 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.474973 9115 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.6:32985
I20240501 19:56:06.475008 9243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.6:32985 every 8 connection(s)
I20240501 19:56:06.476121 9115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:06.479837 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 9115
I20240501 19:56:06.480180 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.7:46465
--local_ip_for_outbound_sockets=127.5.40.7
--tserver_master_addrs=127.5.40.62:40123
--webserver_port=37059
--webserver_interface=127.5.40.7
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:06.485152 9244 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:06.485378 9244 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:06.485956 9244 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:06.487068 7506 ts_manager.cc:194] Re-registered known tserver with Master: 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6:32985)
I20240501 19:56:06.487879 7506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.6:57089
W20240501 19:56:06.588644 9247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:06.588873 9247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:06.588917 9247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:06.591940 9247 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:06.592005 9247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:06.592087 9247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.7
I20240501 19:56:06.595741 9247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.7:46465
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.40.7
--webserver_port=37059
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.9247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.7
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:06.596660 9247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:06.597798 9247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:06.606318 9252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.606739 9253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.607201 9255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:06.607754 9247 server_base.cc:1034] running on GCE node
I20240501 19:56:06.608139 9247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:06.608665 9247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:06.609810 9247 hybrid_clock.cc:648] HybridClock initialized: now 1714593366609788 us; error 38 us; skew 500 ppm
I20240501 19:56:06.611665 9247 webserver.cc:460] Webserver started at http://127.5.40.7:37059/ using document root <none> and password file <none>
I20240501 19:56:06.612202 9247 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:06.612272 9247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:06.615558 9247 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 19:56:06.617354 9262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.618345 9247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:06.618453 9247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-bz40"
I20240501 19:56:06.618867 9247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:06.632902 9247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:06.633538 9247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:06.633689 9247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:06.634215 9247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:06.635104 9247 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:06.635150 9247 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.635201 9247 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:06.635228 9247 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.643651 9247 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.7:46465
I20240501 19:56:06.643682 9375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.7:46465 every 8 connection(s)
I20240501 19:56:06.644786 9247 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:06.645659 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 9247
I20240501 19:56:06.654075 9376 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:06.654320 9376 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:06.654884 9376 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:06.656005 7506 ts_manager.cc:194] Re-registered known tserver with Master: b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7:46465)
I20240501 19:56:06.656741 7506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.7:51309
I20240501 19:56:06.839289 7765 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:06.842648 9178 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.844753 8905 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:06.844822 7631 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:06.847719 8772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.855957 9046 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.865084 9310 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.950537 8838 heartbeater.cc:503] Master 127.5.40.62:40123 was elected leader, sending a full tablet report...
I20240501 19:56:06.975939 7765 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:06.976799 9046 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.979508 9178 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.986403 8905 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:06.989217 9310 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.991717 8772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.994410 7631 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:07.032603 7652 consensus_queue.cc:237] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } }
I20240501 19:56:07.034116 9468 raft_consensus.cc:2932] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } } }
I20240501 19:56:07.036850 7504 catalog_manager.cc:5484] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index -1 to 4, NON_VOTER 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) added. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:07.039353 7585 consensus_peers.cc:485] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 -> Peer 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3:41917): Couldn't send request to peer 29cb7a4d71c44e58956f1ea5afa2e2c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3f595797050475f8e8003c3cdf6d96a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.044263 7785 consensus_queue.cc:237] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } }
I20240501 19:56:07.045820 9479 raft_consensus.cc:2932] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } } }
I20240501 19:56:07.048785 7504 catalog_manager.cc:5484] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index -1 to 4, NON_VOTER b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:07.051241 7718 consensus_peers.cc:485] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 -> Peer b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7:46465): Couldn't send request to peer b39e25d5bdc047c3ac97e42fa43d2aa5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75f7f2b5acf54c43b00fec9c033aadc9. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.055665 7652 consensus_queue.cc:237] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } }
I20240501 19:56:07.056680 9468 raft_consensus.cc:2932] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } } }
I20240501 19:56:07.059137 7504 catalog_manager.cc:5484] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index -1 to 4, NON_VOTER 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:07.061352 7584 consensus_peers.cc:485] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 -> Peer 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5:40713): Couldn't send request to peer 34439dd6bd754595b7219fe0a134e1eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a5b8c745ecf48a083e700b077ce10ce. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.066454 7785 consensus_queue.cc:237] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } }
I20240501 19:56:07.068065 9479 raft_consensus.cc:2932] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } } }
I20240501 19:56:07.071621 7504 catalog_manager.cc:5484] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index -1 to 4, NON_VOTER 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:07.073081 7719 consensus_peers.cc:485] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 -> Peer 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6:32985): Couldn't send request to peer 9d482feaacda4c8ab604c21f0c4b718b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb4733c007f34c4a850f9ac32a778b91. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.080047 7652 consensus_queue.cc:237] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: true } }
I20240501 19:56:07.081066 9468 raft_consensus.cc:2932] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: true } } }
I20240501 19:56:07.083335 7504 catalog_manager.cc:5484] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index -1 to 4, NON_VOTER fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4) added. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:07.086174 7585 consensus_peers.cc:485] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 -> Peer fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4:34385): Couldn't send request to peer fb4dc8d240984a1c9e05c8a99afe2243. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4846083047b478a977ae8e85e0f958d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.091255 7785 consensus_queue.cc:237] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } }
I20240501 19:56:07.092365 9480 raft_consensus.cc:2932] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } } }
I20240501 19:56:07.095110 7509 catalog_manager.cc:5484] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index -1 to 4, NON_VOTER 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) added. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:07.096940 7718 consensus_peers.cc:485] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 -> Peer 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3:41917): Couldn't send request to peer 29cb7a4d71c44e58956f1ea5afa2e2c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b1a8579e6d24bdb8bee0c7fd3fd480b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.102051 7652 consensus_queue.cc:237] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } }
I20240501 19:56:07.103030 9468 raft_consensus.cc:2932] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } } }
W20240501 19:56:07.106559 7585 consensus_peers.cc:485] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 -> Peer b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7:46465): Couldn't send request to peer b39e25d5bdc047c3ac97e42fa43d2aa5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2458bf9cc6ae4ce181f9073df78ee70d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.106544 7509 catalog_manager.cc:5484] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index -1 to 4, NON_VOTER b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:07.112277 7652 consensus_queue.cc:237] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } }
I20240501 19:56:07.113184 9467 raft_consensus.cc:2932] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } } }
W20240501 19:56:07.113670 7584 consensus_peers.cc:485] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 -> Peer 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5:40713): Couldn't send request to peer 34439dd6bd754595b7219fe0a134e1eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b77e626fdd047dab6bbb0a271ada708. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.115464 7509 catalog_manager.cc:5484] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index -1 to 4, NON_VOTER 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:07.123387 7785 consensus_queue.cc:237] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } }
I20240501 19:56:07.124295 9479 raft_consensus.cc:2932] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } } }
W20240501 19:56:07.124761 7719 consensus_peers.cc:485] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 -> Peer 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6:32985): Couldn't send request to peer 9d482feaacda4c8ab604c21f0c4b718b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b567cf2477443bbb17f0c7701fa1411. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.126592 7504 catalog_manager.cc:5484] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index -1 to 4, NON_VOTER 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:07.308424 9112 heartbeater.cc:503] Master 127.5.40.62:40123 was elected leader, sending a full tablet report...
I20240501 19:56:07.489841 9244 heartbeater.cc:503] Master 127.5.40.62:40123 was elected leader, sending a full tablet report...
I20240501 19:56:07.505029 9617 ts_tablet_manager.cc:902] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: Initiating tablet copy from peer 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:07.505724 9617 tablet_copy_client.cc:308] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:39699
I20240501 19:56:07.509372 7804 tablet_copy_service.cc:140] P 09e60f5cde5644f78e51390fce744186: Received BeginTabletCopySession request for tablet 3b567cf2477443bbb17f0c7701fa1411 from peer 9d482feaacda4c8ab604c21f0c4b718b ({username='slave'} at 127.5.40.6:43505)
I20240501 19:56:07.509507 7804 tablet_copy_service.cc:161] P 09e60f5cde5644f78e51390fce744186: Beginning new tablet copy session on tablet 3b567cf2477443bbb17f0c7701fa1411 from peer 9d482feaacda4c8ab604c21f0c4b718b at {username='slave'} at 127.5.40.6:43505: session id = 9d482feaacda4c8ab604c21f0c4b718b-3b567cf2477443bbb17f0c7701fa1411
I20240501 19:56:07.510643 7804 tablet_copy_source_session.cc:205] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.513715 9617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b567cf2477443bbb17f0c7701fa1411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.518836 9617 tablet_copy_client.cc:783] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.519028 9617 tablet_copy_client.cc:647] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.520728 9617 tablet_copy_client.cc:523] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.522361 9617 tablet_bootstrap.cc:492] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: Bootstrap starting.
I20240501 19:56:07.527474 9617 log.cc:824] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:07.546388 9624 ts_tablet_manager.cc:902] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: Initiating tablet copy from peer 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:07.546837 9617 tablet_bootstrap.cc:492] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:07.547240 9617 tablet_bootstrap.cc:492] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: Bootstrap complete.
I20240501 19:56:07.547240 9624 tablet_copy_client.cc:308] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:43301
I20240501 19:56:07.547601 9617 ts_tablet_manager.cc:1367] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.004s
I20240501 19:56:07.550087 9617 raft_consensus.cc:348] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } }
I20240501 19:56:07.550295 9617 raft_consensus.cc:729] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d482feaacda4c8ab604c21f0c4b718b, State: Initialized, Role: LEARNER
I20240501 19:56:07.550647 9617 consensus_queue.cc:260] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } }
I20240501 19:56:07.551467 7672 tablet_copy_service.cc:140] P 5fc8da5aaa7b4840bca58608368e0466: Received BeginTabletCopySession request for tablet d3f595797050475f8e8003c3cdf6d96a from peer 29cb7a4d71c44e58956f1ea5afa2e2c4 ({username='slave'} at 127.5.40.3:44219)
I20240501 19:56:07.551623 7672 tablet_copy_service.cc:161] P 5fc8da5aaa7b4840bca58608368e0466: Beginning new tablet copy session on tablet d3f595797050475f8e8003c3cdf6d96a from peer 29cb7a4d71c44e58956f1ea5afa2e2c4 at {username='slave'} at 127.5.40.3:44219: session id = 29cb7a4d71c44e58956f1ea5afa2e2c4-d3f595797050475f8e8003c3cdf6d96a
I20240501 19:56:07.552997 7672 tablet_copy_source_session.cc:205] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.553992 9617 ts_tablet_manager.cc:1398] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 19:56:07.554731 7804 tablet_copy_service.cc:342] P 09e60f5cde5644f78e51390fce744186: Request end of tablet copy session 9d482feaacda4c8ab604c21f0c4b718b-3b567cf2477443bbb17f0c7701fa1411 received from {username='slave'} at 127.5.40.6:43505
I20240501 19:56:07.554833 7804 tablet_copy_service.cc:433] P 09e60f5cde5644f78e51390fce744186: ending tablet copy session 9d482feaacda4c8ab604c21f0c4b718b-3b567cf2477443bbb17f0c7701fa1411 on tablet 3b567cf2477443bbb17f0c7701fa1411 with peer 9d482feaacda4c8ab604c21f0c4b718b
I20240501 19:56:07.556447 9624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f595797050475f8e8003c3cdf6d96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.561635 9624 tablet_copy_client.cc:783] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.561803 9624 tablet_copy_client.cc:647] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.563416 9624 tablet_copy_client.cc:523] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.564890 9624 tablet_bootstrap.cc:492] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: Bootstrap starting.
I20240501 19:56:07.569980 9624 log.cc:824] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:07.588193 9632 ts_tablet_manager.cc:902] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: Initiating tablet copy from peer 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:07.588909 9632 tablet_copy_client.cc:308] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:43301
I20240501 19:56:07.589629 9624 tablet_bootstrap.cc:492] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:07.590003 9624 tablet_bootstrap.cc:492] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: Bootstrap complete.
I20240501 19:56:07.590341 9624 ts_tablet_manager.cc:1367] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240501 19:56:07.590858 9638 ts_tablet_manager.cc:902] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: Initiating tablet copy from peer 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:07.591553 9638 tablet_copy_client.cc:308] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:43301
I20240501 19:56:07.592936 9624 raft_consensus.cc:348] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } }
I20240501 19:56:07.593209 9624 raft_consensus.cc:729] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29cb7a4d71c44e58956f1ea5afa2e2c4, State: Initialized, Role: LEARNER
I20240501 19:56:07.593248 7672 tablet_copy_service.cc:140] P 5fc8da5aaa7b4840bca58608368e0466: Received BeginTabletCopySession request for tablet e4846083047b478a977ae8e85e0f958d from peer fb4dc8d240984a1c9e05c8a99afe2243 ({username='slave'} at 127.5.40.4:55805)
I20240501 19:56:07.593348 7672 tablet_copy_service.cc:161] P 5fc8da5aaa7b4840bca58608368e0466: Beginning new tablet copy session on tablet e4846083047b478a977ae8e85e0f958d from peer fb4dc8d240984a1c9e05c8a99afe2243 at {username='slave'} at 127.5.40.4:55805: session id = fb4dc8d240984a1c9e05c8a99afe2243-e4846083047b478a977ae8e85e0f958d
I20240501 19:56:07.593587 9624 consensus_queue.cc:260] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } }
I20240501 19:56:07.594271 7672 tablet_copy_source_session.cc:205] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.594475 9624 ts_tablet_manager.cc:1398] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240501 19:56:07.595989 7672 tablet_copy_service.cc:342] P 5fc8da5aaa7b4840bca58608368e0466: Request end of tablet copy session 29cb7a4d71c44e58956f1ea5afa2e2c4-d3f595797050475f8e8003c3cdf6d96a received from {username='slave'} at 127.5.40.3:44219
I20240501 19:56:07.596084 7672 tablet_copy_service.cc:433] P 5fc8da5aaa7b4840bca58608368e0466: ending tablet copy session 29cb7a4d71c44e58956f1ea5afa2e2c4-d3f595797050475f8e8003c3cdf6d96a on tablet d3f595797050475f8e8003c3cdf6d96a with peer 29cb7a4d71c44e58956f1ea5afa2e2c4
I20240501 19:56:07.596621 9646 ts_tablet_manager.cc:902] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: Initiating tablet copy from peer 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:07.597074 9632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4846083047b478a977ae8e85e0f958d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.597373 9646 tablet_copy_client.cc:308] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:39699
I20240501 19:56:07.601073 9632 tablet_copy_client.cc:783] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.601244 9632 tablet_copy_client.cc:647] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.601747 7804 tablet_copy_service.cc:140] P 09e60f5cde5644f78e51390fce744186: Received BeginTabletCopySession request for tablet 75f7f2b5acf54c43b00fec9c033aadc9 from peer b39e25d5bdc047c3ac97e42fa43d2aa5 ({username='slave'} at 127.5.40.7:42755)
I20240501 19:56:07.601855 7804 tablet_copy_service.cc:161] P 09e60f5cde5644f78e51390fce744186: Beginning new tablet copy session on tablet 75f7f2b5acf54c43b00fec9c033aadc9 from peer b39e25d5bdc047c3ac97e42fa43d2aa5 at {username='slave'} at 127.5.40.7:42755: session id = b39e25d5bdc047c3ac97e42fa43d2aa5-75f7f2b5acf54c43b00fec9c033aadc9
I20240501 19:56:07.602677 7804 tablet_copy_source_session.cc:205] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.602768 9632 tablet_copy_client.cc:523] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.604023 9632 tablet_bootstrap.cc:492] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: Bootstrap starting.
I20240501 19:56:07.606034 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f7f2b5acf54c43b00fec9c033aadc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.612522 7672 tablet_copy_service.cc:140] P 5fc8da5aaa7b4840bca58608368e0466: Received BeginTabletCopySession request for tablet 9a5b8c745ecf48a083e700b077ce10ce from peer 34439dd6bd754595b7219fe0a134e1eb ({username='slave'} at 127.5.40.5:47937)
I20240501 19:56:07.612625 9646 tablet_copy_client.cc:783] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.612648 7672 tablet_copy_service.cc:161] P 5fc8da5aaa7b4840bca58608368e0466: Beginning new tablet copy session on tablet 9a5b8c745ecf48a083e700b077ce10ce from peer 34439dd6bd754595b7219fe0a134e1eb at {username='slave'} at 127.5.40.5:47937: session id = 34439dd6bd754595b7219fe0a134e1eb-9a5b8c745ecf48a083e700b077ce10ce
I20240501 19:56:07.612807 9646 tablet_copy_client.cc:647] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.613550 7672 tablet_copy_source_session.cc:205] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.614344 9646 tablet_copy_client.cc:523] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.616271 9646 tablet_bootstrap.cc:492] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: Bootstrap starting.
I20240501 19:56:07.616966 9638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5b8c745ecf48a083e700b077ce10ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.622763 9646 log.cc:824] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:07.623807 9638 tablet_copy_client.cc:783] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.623984 9638 tablet_copy_client.cc:647] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.625995 9638 tablet_copy_client.cc:523] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.627197 9632 tablet_bootstrap.cc:492] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:07.627480 9632 tablet_bootstrap.cc:492] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: Bootstrap complete.
I20240501 19:56:07.627651 9632 ts_tablet_manager.cc:1367] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.006s
I20240501 19:56:07.627992 9638 tablet_bootstrap.cc:492] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: Bootstrap starting.
I20240501 19:56:07.628268 9632 raft_consensus.cc:348] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: true } }
I20240501 19:56:07.628396 9632 raft_consensus.cc:729] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb4dc8d240984a1c9e05c8a99afe2243, State: Initialized, Role: LEARNER
I20240501 19:56:07.628507 9632 consensus_queue.cc:260] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: true } }
I20240501 19:56:07.628857 9632 ts_tablet_manager.cc:1398] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.629428 7672 tablet_copy_service.cc:342] P 5fc8da5aaa7b4840bca58608368e0466: Request end of tablet copy session fb4dc8d240984a1c9e05c8a99afe2243-e4846083047b478a977ae8e85e0f958d received from {username='slave'} at 127.5.40.4:55805
I20240501 19:56:07.629513 7672 tablet_copy_service.cc:433] P 5fc8da5aaa7b4840bca58608368e0466: ending tablet copy session fb4dc8d240984a1c9e05c8a99afe2243-e4846083047b478a977ae8e85e0f958d on tablet e4846083047b478a977ae8e85e0f958d with peer fb4dc8d240984a1c9e05c8a99afe2243
I20240501 19:56:07.635211 9638 log.cc:824] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:07.643308 9617 ts_tablet_manager.cc:902] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: Initiating tablet copy from peer 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:07.643815 9617 tablet_copy_client.cc:308] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:39699
I20240501 19:56:07.644331 7804 tablet_copy_service.cc:140] P 09e60f5cde5644f78e51390fce744186: Received BeginTabletCopySession request for tablet eb4733c007f34c4a850f9ac32a778b91 from peer 9d482feaacda4c8ab604c21f0c4b718b ({username='slave'} at 127.5.40.6:43505)
I20240501 19:56:07.644430 7804 tablet_copy_service.cc:161] P 09e60f5cde5644f78e51390fce744186: Beginning new tablet copy session on tablet eb4733c007f34c4a850f9ac32a778b91 from peer 9d482feaacda4c8ab604c21f0c4b718b at {username='slave'} at 127.5.40.6:43505: session id = 9d482feaacda4c8ab604c21f0c4b718b-eb4733c007f34c4a850f9ac32a778b91
I20240501 19:56:07.645364 7804 tablet_copy_source_session.cc:205] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.647020 9617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4733c007f34c4a850f9ac32a778b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.649078 9617 tablet_copy_client.cc:783] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.649201 9617 tablet_copy_client.cc:647] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.650364 9617 tablet_copy_client.cc:523] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.651589 9617 tablet_bootstrap.cc:492] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: Bootstrap starting.
I20240501 19:56:07.651747 9670 ts_tablet_manager.cc:902] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: Initiating tablet copy from peer 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:07.652302 9670 tablet_copy_client.cc:308] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:43301
I20240501 19:56:07.657330 9646 tablet_bootstrap.cc:492] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:07.657675 9646 tablet_bootstrap.cc:492] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: Bootstrap complete.
I20240501 19:56:07.658210 9646 ts_tablet_manager.cc:1367] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: Time spent bootstrapping tablet: real 0.042s user 0.032s sys 0.004s
I20240501 19:56:07.658712 9376 heartbeater.cc:503] Master 127.5.40.62:40123 was elected leader, sending a full tablet report...
I20240501 19:56:07.660961 7672 tablet_copy_service.cc:140] P 5fc8da5aaa7b4840bca58608368e0466: Received BeginTabletCopySession request for tablet 2458bf9cc6ae4ce181f9073df78ee70d from peer b39e25d5bdc047c3ac97e42fa43d2aa5 ({username='slave'} at 127.5.40.7:38383)
I20240501 19:56:07.661068 7672 tablet_copy_service.cc:161] P 5fc8da5aaa7b4840bca58608368e0466: Beginning new tablet copy session on tablet 2458bf9cc6ae4ce181f9073df78ee70d from peer b39e25d5bdc047c3ac97e42fa43d2aa5 at {username='slave'} at 127.5.40.7:38383: session id = b39e25d5bdc047c3ac97e42fa43d2aa5-2458bf9cc6ae4ce181f9073df78ee70d
I20240501 19:56:07.662003 7672 tablet_copy_source_session.cc:205] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.662858 9670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2458bf9cc6ae4ce181f9073df78ee70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.665144 9670 tablet_copy_client.cc:783] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.665344 9670 tablet_copy_client.cc:647] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.665707 9646 raft_consensus.cc:348] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } }
I20240501 19:56:07.665973 9646 raft_consensus.cc:729] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b39e25d5bdc047c3ac97e42fa43d2aa5, State: Initialized, Role: LEARNER
I20240501 19:56:07.666296 9676 ts_tablet_manager.cc:902] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: Initiating tablet copy from peer 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:07.666368 9646 consensus_queue.cc:260] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } }
I20240501 19:56:07.666447 9670 tablet_copy_client.cc:523] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.666909 9676 tablet_copy_client.cc:308] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:43301
I20240501 19:56:07.667358 9646 ts_tablet_manager.cc:1398] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240501 19:56:07.667430 7672 tablet_copy_service.cc:140] P 5fc8da5aaa7b4840bca58608368e0466: Received BeginTabletCopySession request for tablet 9b77e626fdd047dab6bbb0a271ada708 from peer 34439dd6bd754595b7219fe0a134e1eb ({username='slave'} at 127.5.40.5:47937)
I20240501 19:56:07.667547 7672 tablet_copy_service.cc:161] P 5fc8da5aaa7b4840bca58608368e0466: Beginning new tablet copy session on tablet 9b77e626fdd047dab6bbb0a271ada708 from peer 34439dd6bd754595b7219fe0a134e1eb at {username='slave'} at 127.5.40.5:47937: session id = 34439dd6bd754595b7219fe0a134e1eb-9b77e626fdd047dab6bbb0a271ada708
I20240501 19:56:07.667853 9670 tablet_bootstrap.cc:492] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: Bootstrap starting.
I20240501 19:56:07.668413 7672 tablet_copy_source_session.cc:205] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.669344 9676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b77e626fdd047dab6bbb0a271ada708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.669713 7804 tablet_copy_service.cc:342] P 09e60f5cde5644f78e51390fce744186: Request end of tablet copy session b39e25d5bdc047c3ac97e42fa43d2aa5-75f7f2b5acf54c43b00fec9c033aadc9 received from {username='slave'} at 127.5.40.7:42755
I20240501 19:56:07.669800 7804 tablet_copy_service.cc:433] P 09e60f5cde5644f78e51390fce744186: ending tablet copy session b39e25d5bdc047c3ac97e42fa43d2aa5-75f7f2b5acf54c43b00fec9c033aadc9 on tablet 75f7f2b5acf54c43b00fec9c033aadc9 with peer b39e25d5bdc047c3ac97e42fa43d2aa5
I20240501 19:56:07.671458 9676 tablet_copy_client.cc:783] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.671734 9676 tablet_copy_client.cc:647] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.672695 9676 tablet_copy_client.cc:523] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.672753 9638 tablet_bootstrap.cc:492] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:07.673148 9638 tablet_bootstrap.cc:492] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: Bootstrap complete.
I20240501 19:56:07.673700 9638 ts_tablet_manager.cc:1367] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: Time spent bootstrapping tablet: real 0.046s user 0.035s sys 0.001s
I20240501 19:56:07.673874 9676 tablet_bootstrap.cc:492] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: Bootstrap starting.
I20240501 19:56:07.676543 9638 raft_consensus.cc:348] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } }
I20240501 19:56:07.676787 9638 raft_consensus.cc:729] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 34439dd6bd754595b7219fe0a134e1eb, State: Initialized, Role: LEARNER
I20240501 19:56:07.677147 9638 consensus_queue.cc:260] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } }
I20240501 19:56:07.678414 9617 tablet_bootstrap.cc:492] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:07.678650 9617 tablet_bootstrap.cc:492] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: Bootstrap complete.
I20240501 19:56:07.678793 9617 ts_tablet_manager.cc:1367] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.003s
I20240501 19:56:07.678962 9638 ts_tablet_manager.cc:1398] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:07.680562 9617 raft_consensus.cc:348] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } }
I20240501 19:56:07.680677 9617 raft_consensus.cc:729] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d482feaacda4c8ab604c21f0c4b718b, State: Initialized, Role: LEARNER
I20240501 19:56:07.680773 9617 consensus_queue.cc:260] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: true } }
I20240501 19:56:07.681032 9617 ts_tablet_manager.cc:1398] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:07.681319 7672 tablet_copy_service.cc:342] P 5fc8da5aaa7b4840bca58608368e0466: Request end of tablet copy session 34439dd6bd754595b7219fe0a134e1eb-9a5b8c745ecf48a083e700b077ce10ce received from {username='slave'} at 127.5.40.5:47937
I20240501 19:56:07.681409 7672 tablet_copy_service.cc:433] P 5fc8da5aaa7b4840bca58608368e0466: ending tablet copy session 34439dd6bd754595b7219fe0a134e1eb-9a5b8c745ecf48a083e700b077ce10ce on tablet 9a5b8c745ecf48a083e700b077ce10ce with peer 34439dd6bd754595b7219fe0a134e1eb
I20240501 19:56:07.681591 7804 tablet_copy_service.cc:342] P 09e60f5cde5644f78e51390fce744186: Request end of tablet copy session 9d482feaacda4c8ab604c21f0c4b718b-eb4733c007f34c4a850f9ac32a778b91 received from {username='slave'} at 127.5.40.6:43505
I20240501 19:56:07.681666 7804 tablet_copy_service.cc:433] P 09e60f5cde5644f78e51390fce744186: ending tablet copy session 9d482feaacda4c8ab604c21f0c4b718b-eb4733c007f34c4a850f9ac32a778b91 on tablet eb4733c007f34c4a850f9ac32a778b91 with peer 9d482feaacda4c8ab604c21f0c4b718b
I20240501 19:56:07.691114 9624 ts_tablet_manager.cc:902] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: Initiating tablet copy from peer 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:07.691576 9624 tablet_copy_client.cc:308] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:39699
I20240501 19:56:07.696746 9670 tablet_bootstrap.cc:492] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:07.696934 7804 tablet_copy_service.cc:140] P 09e60f5cde5644f78e51390fce744186: Received BeginTabletCopySession request for tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b from peer 29cb7a4d71c44e58956f1ea5afa2e2c4 ({username='slave'} at 127.5.40.3:33561)
I20240501 19:56:07.697028 7804 tablet_copy_service.cc:161] P 09e60f5cde5644f78e51390fce744186: Beginning new tablet copy session on tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b from peer 29cb7a4d71c44e58956f1ea5afa2e2c4 at {username='slave'} at 127.5.40.3:33561: session id = 29cb7a4d71c44e58956f1ea5afa2e2c4-5b1a8579e6d24bdb8bee0c7fd3fd480b
I20240501 19:56:07.697911 7804 tablet_copy_source_session.cc:205] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.698489 9670 tablet_bootstrap.cc:492] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: Bootstrap complete.
I20240501 19:56:07.698747 9670 ts_tablet_manager.cc:1367] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.000s
I20240501 19:56:07.698783 9624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.699256 9670 raft_consensus.cc:348] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } }
I20240501 19:56:07.699343 9676 tablet_bootstrap.cc:492] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:07.699358 9670 raft_consensus.cc:729] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b39e25d5bdc047c3ac97e42fa43d2aa5, State: Initialized, Role: LEARNER
I20240501 19:56:07.699473 9670 consensus_queue.cc:260] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: true } }
I20240501 19:56:07.699636 9676 tablet_bootstrap.cc:492] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: Bootstrap complete.
I20240501 19:56:07.699826 9670 ts_tablet_manager.cc:1398] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.699853 9676 ts_tablet_manager.cc:1367] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240501 19:56:07.700362 7672 tablet_copy_service.cc:342] P 5fc8da5aaa7b4840bca58608368e0466: Request end of tablet copy session b39e25d5bdc047c3ac97e42fa43d2aa5-2458bf9cc6ae4ce181f9073df78ee70d received from {username='slave'} at 127.5.40.7:38383
I20240501 19:56:07.700441 7672 tablet_copy_service.cc:433] P 5fc8da5aaa7b4840bca58608368e0466: ending tablet copy session b39e25d5bdc047c3ac97e42fa43d2aa5-2458bf9cc6ae4ce181f9073df78ee70d on tablet 2458bf9cc6ae4ce181f9073df78ee70d with peer b39e25d5bdc047c3ac97e42fa43d2aa5
I20240501 19:56:07.700939 9624 tablet_copy_client.cc:783] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.700953 9676 raft_consensus.cc:348] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } }
I20240501 19:56:07.701059 9676 raft_consensus.cc:729] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 34439dd6bd754595b7219fe0a134e1eb, State: Initialized, Role: LEARNER
I20240501 19:56:07.701078 9624 tablet_copy_client.cc:647] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.701144 9676 consensus_queue.cc:260] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: true } }
I20240501 19:56:07.701395 9676 ts_tablet_manager.cc:1398] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.701874 7672 tablet_copy_service.cc:342] P 5fc8da5aaa7b4840bca58608368e0466: Request end of tablet copy session 34439dd6bd754595b7219fe0a134e1eb-9b77e626fdd047dab6bbb0a271ada708 received from {username='slave'} at 127.5.40.5:47937
I20240501 19:56:07.701946 7672 tablet_copy_service.cc:433] P 5fc8da5aaa7b4840bca58608368e0466: ending tablet copy session 34439dd6bd754595b7219fe0a134e1eb-9b77e626fdd047dab6bbb0a271ada708 on tablet 9b77e626fdd047dab6bbb0a271ada708 with peer 34439dd6bd754595b7219fe0a134e1eb
I20240501 19:56:07.702880 9624 tablet_copy_client.cc:523] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:07.704135 9624 tablet_bootstrap.cc:492] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: Bootstrap starting.
I20240501 19:56:07.724603 9624 tablet_bootstrap.cc:492] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:07.724893 9624 tablet_bootstrap.cc:492] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: Bootstrap complete.
I20240501 19:56:07.725077 9624 ts_tablet_manager.cc:1367] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20240501 19:56:07.725701 9624 raft_consensus.cc:348] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } }
I20240501 19:56:07.725811 9624 raft_consensus.cc:729] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29cb7a4d71c44e58956f1ea5afa2e2c4, State: Initialized, Role: LEARNER
I20240501 19:56:07.725903 9624 consensus_queue.cc:260] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: true } }
I20240501 19:56:07.726173 9624 ts_tablet_manager.cc:1398] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.726790 7804 tablet_copy_service.cc:342] P 09e60f5cde5644f78e51390fce744186: Request end of tablet copy session 29cb7a4d71c44e58956f1ea5afa2e2c4-5b1a8579e6d24bdb8bee0c7fd3fd480b received from {username='slave'} at 127.5.40.3:33561
I20240501 19:56:07.726879 7804 tablet_copy_service.cc:433] P 09e60f5cde5644f78e51390fce744186: ending tablet copy session 29cb7a4d71c44e58956f1ea5afa2e2c4-5b1a8579e6d24bdb8bee0c7fd3fd480b on tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b with peer 29cb7a4d71c44e58956f1ea5afa2e2c4
I20240501 19:56:07.993011 9198 raft_consensus.cc:1203] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.114317 9066 raft_consensus.cc:1203] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.118721 9330 raft_consensus.cc:1203] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.120246 8792 raft_consensus.cc:1203] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.161954 8925 raft_consensus.cc:1203] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.168617 9330 raft_consensus.cc:1203] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.176138 9198 raft_consensus.cc:1203] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.222904 8792 raft_consensus.cc:1203] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.281713 9066 raft_consensus.cc:1203] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:08.445943 9772 raft_consensus.cc:1050] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: attempting to promote NON_VOTER 9d482feaacda4c8ab604c21f0c4b718b to VOTER
I20240501 19:56:08.446462 9772 consensus_queue.cc:237] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:08.447727 9198 raft_consensus.cc:1261] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 LEARNER]: Refusing update from remote peer 09e60f5cde5644f78e51390fce744186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:08.448194 9773 consensus_queue.cc:1035] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:08.451598 9772 raft_consensus.cc:2932] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } } }
I20240501 19:56:08.452378 9198 raft_consensus.cc:2932] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } } }
I20240501 19:56:08.454332 7509 catalog_manager.cc:5484] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index 4 to 5, 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.555146 9526 raft_consensus.cc:1050] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: attempting to promote NON_VOTER b39e25d5bdc047c3ac97e42fa43d2aa5 to VOTER
I20240501 19:56:08.555521 9526 consensus_queue.cc:237] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:08.556681 9330 raft_consensus.cc:1261] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 LEARNER]: Refusing update from remote peer 09e60f5cde5644f78e51390fce744186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:08.557071 9526 consensus_queue.cc:1035] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:08.559585 9773 raft_consensus.cc:2932] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } } }
I20240501 19:56:08.560205 9330 raft_consensus.cc:2932] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } } }
I20240501 19:56:08.562325 7504 catalog_manager.cc:5484] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index 4 to 5, b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.571691 9774 raft_consensus.cc:1050] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: attempting to promote NON_VOTER 29cb7a4d71c44e58956f1ea5afa2e2c4 to VOTER
I20240501 19:56:08.572198 9774 consensus_queue.cc:237] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:08.573360 8792 raft_consensus.cc:1261] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 LEARNER]: Refusing update from remote peer 5fc8da5aaa7b4840bca58608368e0466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:08.573834 9776 consensus_queue.cc:1035] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:08.576445 9774 raft_consensus.cc:2932] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } } }
I20240501 19:56:08.577003 8792 raft_consensus.cc:2932] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } } }
I20240501 19:56:08.578795 9774 raft_consensus.cc:1050] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: attempting to promote NON_VOTER b39e25d5bdc047c3ac97e42fa43d2aa5 to VOTER
I20240501 19:56:08.579121 9774 consensus_queue.cc:237] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:08.579378 7509 catalog_manager.cc:5484] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 reported cstate change: config changed from index 4 to 5, 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } } }
I20240501 19:56:08.580157 9330 raft_consensus.cc:1261] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 LEARNER]: Refusing update from remote peer 5fc8da5aaa7b4840bca58608368e0466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:08.580520 9776 consensus_queue.cc:1035] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:08.582343 9774 raft_consensus.cc:2932] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } } }
I20240501 19:56:08.582798 9330 raft_consensus.cc:2932] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } } }
I20240501 19:56:08.584934 7504 catalog_manager.cc:5484] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 reported cstate change: config changed from index 4 to 5, b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } } }
I20240501 19:56:08.600227 9775 raft_consensus.cc:1050] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: attempting to promote NON_VOTER 34439dd6bd754595b7219fe0a134e1eb to VOTER
I20240501 19:56:08.600566 9775 consensus_queue.cc:237] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:08.601871 9066 raft_consensus.cc:1261] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 1 LEARNER]: Refusing update from remote peer 5fc8da5aaa7b4840bca58608368e0466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:08.602289 9775 consensus_queue.cc:1035] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:08.604907 9774 raft_consensus.cc:2932] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } } }
I20240501 19:56:08.605437 9066 raft_consensus.cc:2932] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } } }
I20240501 19:56:08.607211 7509 catalog_manager.cc:5484] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index 4 to 5, 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.608592 9526 raft_consensus.cc:1050] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: attempting to promote NON_VOTER 9d482feaacda4c8ab604c21f0c4b718b to VOTER
I20240501 19:56:08.608919 9526 consensus_queue.cc:237] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:08.610059 9198 raft_consensus.cc:1261] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 LEARNER]: Refusing update from remote peer 09e60f5cde5644f78e51390fce744186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:08.610421 9773 consensus_queue.cc:1035] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:08.612066 9526 raft_consensus.cc:2932] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } } }
I20240501 19:56:08.612546 9198 raft_consensus.cc:2932] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } } }
I20240501 19:56:08.614432 7504 catalog_manager.cc:5484] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index 4 to 5, 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.693650 9774 raft_consensus.cc:1050] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: attempting to promote NON_VOTER fb4dc8d240984a1c9e05c8a99afe2243 to VOTER
I20240501 19:56:08.694017 9774 consensus_queue.cc:237] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } }
I20240501 19:56:08.695154 8925 raft_consensus.cc:1261] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 LEARNER]: Refusing update from remote peer 5fc8da5aaa7b4840bca58608368e0466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:08.695624 9774 consensus_queue.cc:1035] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:08.697698 9776 raft_consensus.cc:2932] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } } }
I20240501 19:56:08.698338 8925 raft_consensus.cc:2932] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } } }
I20240501 19:56:08.700259 7509 catalog_manager.cc:5484] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index 4 to 5, fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.706120 7785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3b567cf2477443bbb17f0c7701fa1411"
dest_uuid: "09e60f5cde5644f78e51390fce744186"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57732
I20240501 19:56:08.706296 7785 raft_consensus.cc:595] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:08.715996 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2458bf9cc6ae4ce181f9073df78ee70d"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48548
I20240501 19:56:08.716178 7652 raft_consensus.cc:595] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:08.726040 7785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75f7f2b5acf54c43b00fec9c033aadc9"
dest_uuid: "09e60f5cde5644f78e51390fce744186"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57756
I20240501 19:56:08.726183 7785 raft_consensus.cc:595] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:08.736728 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a5b8c745ecf48a083e700b077ce10ce"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48576
I20240501 19:56:08.736876 7652 raft_consensus.cc:595] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:08.747150 7785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb4733c007f34c4a850f9ac32a778b91"
dest_uuid: "09e60f5cde5644f78e51390fce744186"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57766
I20240501 19:56:08.747299 7785 raft_consensus.cc:595] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:08.752969 9774 raft_consensus.cc:1050] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: attempting to promote NON_VOTER 34439dd6bd754595b7219fe0a134e1eb to VOTER
I20240501 19:56:08.753337 9774 consensus_queue.cc:237] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:08.754508 9066 raft_consensus.cc:1261] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 1 LEARNER]: Refusing update from remote peer 5fc8da5aaa7b4840bca58608368e0466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:08.754925 9775 consensus_queue.cc:1035] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:08.756606 9774 raft_consensus.cc:2932] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } } }
I20240501 19:56:08.757083 9066 raft_consensus.cc:2932] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } } }
I20240501 19:56:08.757992 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3f595797050475f8e8003c3cdf6d96a"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48596
I20240501 19:56:08.758090 7652 raft_consensus.cc:595] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:08.759308 7504 catalog_manager.cc:5484] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index 4 to 5, 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.776749 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4846083047b478a977ae8e85e0f958d"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48616
I20240501 19:56:08.776913 7652 raft_consensus.cc:595] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:08.810052 9773 raft_consensus.cc:1050] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: attempting to promote NON_VOTER 29cb7a4d71c44e58956f1ea5afa2e2c4 to VOTER
I20240501 19:56:08.810416 9773 consensus_queue.cc:237] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:08.811568 8792 raft_consensus.cc:1261] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 LEARNER]: Refusing update from remote peer 09e60f5cde5644f78e51390fce744186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:08.811987 9773 consensus_queue.cc:1035] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:08.813637 9526 raft_consensus.cc:2932] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } } }
I20240501 19:56:08.814074 8792 raft_consensus.cc:2932] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } } }
I20240501 19:56:08.816058 7509 catalog_manager.cc:5484] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index 4 to 5, 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.932417 9772 raft_consensus.cc:979] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: : Instructing follower 9d482feaacda4c8ab604c21f0c4b718b to start an election
I20240501 19:56:08.932539 9772 raft_consensus.cc:1067] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Signalling peer 9d482feaacda4c8ab604c21f0c4b718b to start an election
I20240501 19:56:08.933094 9198 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b567cf2477443bbb17f0c7701fa1411"
dest_uuid: "9d482feaacda4c8ab604c21f0c4b718b"
from {username='slave'} at 127.5.40.2:55521
I20240501 19:56:08.933274 9198 raft_consensus.cc:482] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:08.933355 9198 raft_consensus.cc:3037] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:08.934147 9198 raft_consensus.cc:504] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:08.934940 9198 leader_election.cc:288] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [CANDIDATE]: Term 2 election: Requested vote from peers 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:08.935370 7785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b567cf2477443bbb17f0c7701fa1411" candidate_uuid: "9d482feaacda4c8ab604c21f0c4b718b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09e60f5cde5644f78e51390fce744186"
I20240501 19:56:08.935559 7785 raft_consensus.cc:3032] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:08.935601 7785 raft_consensus.cc:729] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:08.935746 7785 consensus_queue.cc:260] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:08.935897 7785 raft_consensus.cc:3037] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:08.936789 7785 raft_consensus.cc:2454] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d482feaacda4c8ab604c21f0c4b718b in term 2.
I20240501 19:56:08.937183 9134 leader_election.cc:302] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186, 9d482feaacda4c8ab604c21f0c4b718b; no voters:
I20240501 19:56:08.937438 9630 raft_consensus.cc:2781] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:08.937618 9630 raft_consensus.cc:686] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 2 LEADER]: Becoming Leader. State: Replica: 9d482feaacda4c8ab604c21f0c4b718b, State: Running, Role: LEADER
I20240501 19:56:08.937901 9630 consensus_queue.cc:237] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:08.940760 7504 catalog_manager.cc:5484] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b reported cstate change: term changed from 1 to 2, leader changed from 09e60f5cde5644f78e51390fce744186 (127.5.40.2) to 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6). New cstate: current_term: 2 leader_uuid: "9d482feaacda4c8ab604c21f0c4b718b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 19:56:08.947106 7495 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 3b567cf2477443bbb17f0c7701fa1411 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6:32985) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 19:56:08.956022 9776 raft_consensus.cc:979] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: : Instructing follower 29cb7a4d71c44e58956f1ea5afa2e2c4 to start an election
I20240501 19:56:08.956135 9776 raft_consensus.cc:1067] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Signalling peer 29cb7a4d71c44e58956f1ea5afa2e2c4 to start an election
I20240501 19:56:08.956727 8792 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3f595797050475f8e8003c3cdf6d96a"
dest_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4"
from {username='slave'} at 127.5.40.1:46889
I20240501 19:56:08.956902 8792 raft_consensus.cc:482] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:08.956985 8792 raft_consensus.cc:3037] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:08.957757 8792 raft_consensus.cc:504] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:08.958581 8792 leader_election.cc:288] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [CANDIDATE]: Term 2 election: Requested vote from peers 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:08.959053 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f595797050475f8e8003c3cdf6d96a" candidate_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
I20240501 19:56:08.959197 7652 raft_consensus.cc:3032] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:08.959234 7652 raft_consensus.cc:729] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:08.959362 7652 consensus_queue.cc:260] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:08.959560 7652 raft_consensus.cc:3037] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:08.960333 7652 raft_consensus.cc:2454] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb7a4d71c44e58956f1ea5afa2e2c4 in term 2.
I20240501 19:56:08.960700 8724 leader_election.cc:302] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 29cb7a4d71c44e58956f1ea5afa2e2c4, 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:08.960916 9645 raft_consensus.cc:2781] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:08.961088 9645 raft_consensus.cc:686] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 2 LEADER]: Becoming Leader. State: Replica: 29cb7a4d71c44e58956f1ea5afa2e2c4, State: Running, Role: LEADER
I20240501 19:56:08.961346 9645 consensus_queue.cc:237] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:08.963230 7504 catalog_manager.cc:5484] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 reported cstate change: term changed from 1 to 2, leader changed from 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) to 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3). New cstate: current_term: 2 leader_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.988119 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b77e626fdd047dab6bbb0a271ada708"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48622
I20240501 19:56:08.988260 7652 raft_consensus.cc:595] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:09.003091 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2458bf9cc6ae4ce181f9073df78ee70d"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48636
I20240501 19:56:09.003224 7652 raft_consensus.cc:595] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:09.013114 7785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75f7f2b5acf54c43b00fec9c033aadc9"
dest_uuid: "09e60f5cde5644f78e51390fce744186"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57782
I20240501 19:56:09.013263 7785 raft_consensus.cc:595] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:09.015729 9775 raft_consensus.cc:979] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: : Instructing follower b39e25d5bdc047c3ac97e42fa43d2aa5 to start an election
I20240501 19:56:09.015857 9775 raft_consensus.cc:1067] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Signalling peer b39e25d5bdc047c3ac97e42fa43d2aa5 to start an election
I20240501 19:56:09.016366 9330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2458bf9cc6ae4ce181f9073df78ee70d"
dest_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5"
from {username='slave'} at 127.5.40.1:41399
I20240501 19:56:09.016563 9330 raft_consensus.cc:482] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:09.016672 9330 raft_consensus.cc:3037] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.017697 9330 raft_consensus.cc:504] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:09.018985 9330 leader_election.cc:288] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [CANDIDATE]: Term 2 election: Requested vote from peers 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:09.019343 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2458bf9cc6ae4ce181f9073df78ee70d" candidate_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
I20240501 19:56:09.019479 7652 raft_consensus.cc:3032] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:09.019524 7652 raft_consensus.cc:729] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:09.019665 7652 consensus_queue.cc:260] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:09.019804 7652 raft_consensus.cc:3037] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.020539 7652 raft_consensus.cc:2454] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b39e25d5bdc047c3ac97e42fa43d2aa5 in term 2.
I20240501 19:56:09.020926 9263 leader_election.cc:302] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466, b39e25d5bdc047c3ac97e42fa43d2aa5; no voters:
I20240501 19:56:09.021163 9682 raft_consensus.cc:2781] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.021318 9682 raft_consensus.cc:686] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 2 LEADER]: Becoming Leader. State: Replica: b39e25d5bdc047c3ac97e42fa43d2aa5, State: Running, Role: LEADER
I20240501 19:56:09.021622 9682 consensus_queue.cc:237] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:09.023803 7504 catalog_manager.cc:5484] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 reported cstate change: term changed from 1 to 2, leader changed from 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) to b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7). New cstate: current_term: 2 leader_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.025549 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a5b8c745ecf48a083e700b077ce10ce"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48662
I20240501 19:56:09.025679 7652 raft_consensus.cc:595] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:09.036101 7785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb4733c007f34c4a850f9ac32a778b91"
dest_uuid: "09e60f5cde5644f78e51390fce744186"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57798
I20240501 19:56:09.036239 7785 raft_consensus.cc:595] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:09.052186 7785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b1a8579e6d24bdb8bee0c7fd3fd480b"
dest_uuid: "09e60f5cde5644f78e51390fce744186"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57810
I20240501 19:56:09.052320 7785 raft_consensus.cc:595] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:09.062414 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4846083047b478a977ae8e85e0f958d"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48670
I20240501 19:56:09.062556 7652 raft_consensus.cc:595] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:09.121955 9773 raft_consensus.cc:979] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: : Instructing follower b39e25d5bdc047c3ac97e42fa43d2aa5 to start an election
I20240501 19:56:09.122081 9773 raft_consensus.cc:1067] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Signalling peer b39e25d5bdc047c3ac97e42fa43d2aa5 to start an election
I20240501 19:56:09.122614 9330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75f7f2b5acf54c43b00fec9c033aadc9"
dest_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5"
from {username='slave'} at 127.5.40.2:33673
I20240501 19:56:09.122757 9330 raft_consensus.cc:482] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:09.122826 9330 raft_consensus.cc:3037] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.135495 9772 raft_consensus.cc:979] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: : Instructing follower 9d482feaacda4c8ab604c21f0c4b718b to start an election
I20240501 19:56:09.135603 9772 raft_consensus.cc:1067] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Signalling peer 9d482feaacda4c8ab604c21f0c4b718b to start an election
I20240501 19:56:09.136055 9198 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb4733c007f34c4a850f9ac32a778b91"
dest_uuid: "9d482feaacda4c8ab604c21f0c4b718b"
from {username='slave'} at 127.5.40.2:55521
I20240501 19:56:09.136191 9198 raft_consensus.cc:482] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:09.136253 9198 raft_consensus.cc:3037] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.184177 9330 raft_consensus.cc:504] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:09.184177 9198 raft_consensus.cc:504] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:09.184633 9198 leader_election.cc:288] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [CANDIDATE]: Term 2 election: Requested vote from peers 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:09.184670 9330 leader_election.cc:288] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [CANDIDATE]: Term 2 election: Requested vote from peers 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:09.185153 7785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4733c007f34c4a850f9ac32a778b91" candidate_uuid: "9d482feaacda4c8ab604c21f0c4b718b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09e60f5cde5644f78e51390fce744186"
I20240501 19:56:09.185295 7785 raft_consensus.cc:3032] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:09.185344 7785 raft_consensus.cc:729] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:09.185385 7784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f7f2b5acf54c43b00fec9c033aadc9" candidate_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09e60f5cde5644f78e51390fce744186"
I20240501 19:56:09.185521 7784 raft_consensus.cc:3032] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:09.185508 7785 consensus_queue.cc:260] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:09.185576 7784 raft_consensus.cc:729] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:09.185652 7785 raft_consensus.cc:3037] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.185735 7784 consensus_queue.cc:260] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:09.185880 7784 raft_consensus.cc:3037] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.186620 7785 raft_consensus.cc:2454] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d482feaacda4c8ab604c21f0c4b718b in term 2.
I20240501 19:56:09.186805 7784 raft_consensus.cc:2454] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b39e25d5bdc047c3ac97e42fa43d2aa5 in term 2.
I20240501 19:56:09.186986 9134 leader_election.cc:302] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186, 9d482feaacda4c8ab604c21f0c4b718b; no voters:
I20240501 19:56:09.187086 9266 leader_election.cc:302] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186, b39e25d5bdc047c3ac97e42fa43d2aa5; no voters:
I20240501 19:56:09.187201 9630 raft_consensus.cc:2781] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.187274 9682 raft_consensus.cc:2781] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.187290 9630 raft_consensus.cc:686] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 2 LEADER]: Becoming Leader. State: Replica: 9d482feaacda4c8ab604c21f0c4b718b, State: Running, Role: LEADER
I20240501 19:56:09.187361 9682 raft_consensus.cc:686] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 2 LEADER]: Becoming Leader. State: Replica: b39e25d5bdc047c3ac97e42fa43d2aa5, State: Running, Role: LEADER
I20240501 19:56:09.187498 9630 consensus_queue.cc:237] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:09.187517 9682 consensus_queue.cc:237] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:09.189154 7504 catalog_manager.cc:5484] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b reported cstate change: term changed from 1 to 2, leader changed from 09e60f5cde5644f78e51390fce744186 (127.5.40.2) to 9d482feaacda4c8ab604c21f0c4b718b (127.5.40.6). New cstate: current_term: 2 leader_uuid: "9d482feaacda4c8ab604c21f0c4b718b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.189163 7509 catalog_manager.cc:5484] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 reported cstate change: term changed from 1 to 2, leader changed from 09e60f5cde5644f78e51390fce744186 (127.5.40.2) to b39e25d5bdc047c3ac97e42fa43d2aa5 (127.5.40.7). New cstate: current_term: 2 leader_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.209780 9776 raft_consensus.cc:979] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: : Instructing follower 34439dd6bd754595b7219fe0a134e1eb to start an election
I20240501 19:56:09.209915 9776 raft_consensus.cc:1067] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Signalling peer 34439dd6bd754595b7219fe0a134e1eb to start an election
I20240501 19:56:09.210407 9066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a5b8c745ecf48a083e700b077ce10ce"
dest_uuid: "34439dd6bd754595b7219fe0a134e1eb"
from {username='slave'} at 127.5.40.1:52985
I20240501 19:56:09.210592 9066 raft_consensus.cc:482] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:09.210706 9066 raft_consensus.cc:3037] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.259012 9066 raft_consensus.cc:504] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:09.260670 9066 leader_election.cc:288] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [CANDIDATE]: Term 2 election: Requested vote from peers 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:09.261178 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5b8c745ecf48a083e700b077ce10ce" candidate_uuid: "34439dd6bd754595b7219fe0a134e1eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
I20240501 19:56:09.261310 7652 raft_consensus.cc:3032] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:09.261363 7652 raft_consensus.cc:729] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:09.261554 7652 consensus_queue.cc:260] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:09.261693 7652 raft_consensus.cc:3037] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.262596 7652 raft_consensus.cc:2454] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34439dd6bd754595b7219fe0a134e1eb in term 2.
I20240501 19:56:09.262986 8999 leader_election.cc:302] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34439dd6bd754595b7219fe0a134e1eb, 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:09.263356 10062 raft_consensus.cc:2781] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.263543 10062 raft_consensus.cc:686] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 2 LEADER]: Becoming Leader. State: Replica: 34439dd6bd754595b7219fe0a134e1eb, State: Running, Role: LEADER
I20240501 19:56:09.263837 10062 consensus_queue.cc:237] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:09.265795 7509 catalog_manager.cc:5484] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb reported cstate change: term changed from 1 to 2, leader changed from 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) to 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5). New cstate: current_term: 2 leader_uuid: "34439dd6bd754595b7219fe0a134e1eb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.274173 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b77e626fdd047dab6bbb0a271ada708"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48684
I20240501 19:56:09.274338 7652 raft_consensus.cc:595] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:09.302843 9776 raft_consensus.cc:979] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: : Instructing follower fb4dc8d240984a1c9e05c8a99afe2243 to start an election
I20240501 19:56:09.302951 9776 raft_consensus.cc:1067] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Signalling peer fb4dc8d240984a1c9e05c8a99afe2243 to start an election
I20240501 19:56:09.303419 8925 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4846083047b478a977ae8e85e0f958d"
dest_uuid: "fb4dc8d240984a1c9e05c8a99afe2243"
from {username='slave'} at 127.5.40.1:48701
I20240501 19:56:09.303649 8925 raft_consensus.cc:482] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:09.303720 8925 raft_consensus.cc:3037] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.304705 8925 raft_consensus.cc:504] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } }
I20240501 19:56:09.305330 8925 leader_election.cc:288] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [CANDIDATE]: Term 2 election: Requested vote from peers 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:09.305693 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4846083047b478a977ae8e85e0f958d" candidate_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
I20240501 19:56:09.305820 7652 raft_consensus.cc:3032] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:09.305863 7652 raft_consensus.cc:729] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:09.306011 7652 consensus_queue.cc:260] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } }
I20240501 19:56:09.306159 7652 raft_consensus.cc:3037] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.307075 7652 raft_consensus.cc:2454] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4dc8d240984a1c9e05c8a99afe2243 in term 2.
I20240501 19:56:09.307384 8857 leader_election.cc:302] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5fc8da5aaa7b4840bca58608368e0466, fb4dc8d240984a1c9e05c8a99afe2243; no voters:
I20240501 19:56:09.307693 10105 raft_consensus.cc:2781] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.307843 10105 raft_consensus.cc:686] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 2 LEADER]: Becoming Leader. State: Replica: fb4dc8d240984a1c9e05c8a99afe2243, State: Running, Role: LEADER
I20240501 19:56:09.307997 10105 consensus_queue.cc:237] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } }
I20240501 19:56:09.309746 7509 catalog_manager.cc:5484] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 reported cstate change: term changed from 1 to 2, leader changed from 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) to fb4dc8d240984a1c9e05c8a99afe2243 (127.5.40.4). New cstate: current_term: 2 leader_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.319037 7784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b1a8579e6d24bdb8bee0c7fd3fd480b"
dest_uuid: "09e60f5cde5644f78e51390fce744186"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57816
I20240501 19:56:09.319176 7784 raft_consensus.cc:595] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:09.352908 9776 raft_consensus.cc:979] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: : Instructing follower 34439dd6bd754595b7219fe0a134e1eb to start an election
I20240501 19:56:09.353019 9776 raft_consensus.cc:1067] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Signalling peer 34439dd6bd754595b7219fe0a134e1eb to start an election
I20240501 19:56:09.353415 9066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b77e626fdd047dab6bbb0a271ada708"
dest_uuid: "34439dd6bd754595b7219fe0a134e1eb"
from {username='slave'} at 127.5.40.1:52985
I20240501 19:56:09.353540 9066 raft_consensus.cc:482] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:09.353586 9066 raft_consensus.cc:3037] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.354344 9066 raft_consensus.cc:504] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:09.354629 9066 leader_election.cc:288] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [CANDIDATE]: Term 2 election: Requested vote from peers 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:09.355017 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b77e626fdd047dab6bbb0a271ada708" candidate_uuid: "34439dd6bd754595b7219fe0a134e1eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
I20240501 19:56:09.355141 7652 raft_consensus.cc:3032] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:09.355185 7652 raft_consensus.cc:729] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:09.355319 7652 consensus_queue.cc:260] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:09.355469 7652 raft_consensus.cc:3037] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.356391 7652 raft_consensus.cc:2454] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34439dd6bd754595b7219fe0a134e1eb in term 2.
I20240501 19:56:09.356655 8999 leader_election.cc:302] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34439dd6bd754595b7219fe0a134e1eb, 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:09.356834 10062 raft_consensus.cc:2781] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.356922 10062 raft_consensus.cc:686] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 2 LEADER]: Becoming Leader. State: Replica: 34439dd6bd754595b7219fe0a134e1eb, State: Running, Role: LEADER
I20240501 19:56:09.357065 10062 consensus_queue.cc:237] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:09.358594 7509 catalog_manager.cc:5484] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb reported cstate change: term changed from 1 to 2, leader changed from 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) to 34439dd6bd754595b7219fe0a134e1eb (127.5.40.5). New cstate: current_term: 2 leader_uuid: "34439dd6bd754595b7219fe0a134e1eb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.375363 7784 raft_consensus.cc:1261] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Refusing update from remote peer 9d482feaacda4c8ab604c21f0c4b718b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:09.376072 9630 consensus_queue.cc:1035] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.385706 9773 raft_consensus.cc:979] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: : Instructing follower 29cb7a4d71c44e58956f1ea5afa2e2c4 to start an election
I20240501 19:56:09.385825 9773 raft_consensus.cc:1067] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Signalling peer 29cb7a4d71c44e58956f1ea5afa2e2c4 to start an election
I20240501 19:56:09.386234 8792 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b1a8579e6d24bdb8bee0c7fd3fd480b"
dest_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4"
from {username='slave'} at 127.5.40.2:60311
I20240501 19:56:09.386374 8792 raft_consensus.cc:482] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:09.386441 8792 raft_consensus.cc:3037] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.387290 8792 raft_consensus.cc:504] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:09.387681 8792 leader_election.cc:288] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [CANDIDATE]: Term 2 election: Requested vote from peers 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:09.388057 7784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1a8579e6d24bdb8bee0c7fd3fd480b" candidate_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09e60f5cde5644f78e51390fce744186"
I20240501 19:56:09.388165 7784 raft_consensus.cc:3032] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:09.388199 7784 raft_consensus.cc:729] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:09.388311 7784 consensus_queue.cc:260] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:09.388417 7784 raft_consensus.cc:3037] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.389140 7784 raft_consensus.cc:2454] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb7a4d71c44e58956f1ea5afa2e2c4 in term 2.
I20240501 19:56:09.389423 8727 leader_election.cc:302] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186, 29cb7a4d71c44e58956f1ea5afa2e2c4; no voters:
I20240501 19:56:09.389617 9645 raft_consensus.cc:2781] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.389717 9645 raft_consensus.cc:686] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 2 LEADER]: Becoming Leader. State: Replica: 29cb7a4d71c44e58956f1ea5afa2e2c4, State: Running, Role: LEADER
I20240501 19:56:09.389850 9645 consensus_queue.cc:237] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:09.391461 7504 catalog_manager.cc:5484] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 reported cstate change: term changed from 1 to 2, leader changed from 09e60f5cde5644f78e51390fce744186 (127.5.40.2) to 29cb7a4d71c44e58956f1ea5afa2e2c4 (127.5.40.3). New cstate: current_term: 2 leader_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.393690 7652 raft_consensus.cc:1261] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Refusing update from remote peer 29cb7a4d71c44e58956f1ea5afa2e2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:09.394331 9645 consensus_queue.cc:1035] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.595829 9198 consensus_queue.cc:237] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:09.596555 9630 raft_consensus.cc:2932] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } } }
I20240501 19:56:09.597990 7495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3b567cf2477443bbb17f0c7701fa1411 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 19:56:09.598832 7509 catalog_manager.cc:5484] T 3b567cf2477443bbb17f0c7701fa1411 P 9d482feaacda4c8ab604c21f0c4b718b reported cstate change: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New cstate: current_term: 2 leader_uuid: "9d482feaacda4c8ab604c21f0c4b718b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.601377 7765 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b567cf2477443bbb17f0c7701fa1411 with delete_type TABLET_DATA_TOMBSTONED (TS 09e60f5cde5644f78e51390fce744186 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57340
I20240501 19:56:09.601912 10173 tablet_replica.cc:330] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: stopping tablet replica
I20240501 19:56:09.602236 10173 raft_consensus.cc:2229] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:09.602391 10173 raft_consensus.cc:2258] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.603376 10173 ts_tablet_manager.cc:1875] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:09.604600 7652 raft_consensus.cc:1261] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Refusing update from remote peer b39e25d5bdc047c3ac97e42fa43d2aa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:09.605124 9682 consensus_queue.cc:1035] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.605676 10173 ts_tablet_manager.cc:1888] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:09.605765 10173 log.cc:1196] T 3b567cf2477443bbb17f0c7701fa1411 P 09e60f5cde5644f78e51390fce744186: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/3b567cf2477443bbb17f0c7701fa1411
I20240501 19:56:09.606781 7496 catalog_manager.cc:4830] TS 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699): tablet 3b567cf2477443bbb17f0c7701fa1411 (table rebalance_test_table_3 [id=deba21d5b67a4c8a9b2322eca8b8faf1]) successfully deleted
I20240501 19:56:09.630724 8792 consensus_queue.cc:237] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:09.631374 9645 raft_consensus.cc:2932] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } } }
I20240501 19:56:09.632653 7494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3f595797050475f8e8003c3cdf6d96a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 19:56:09.633504 7504 catalog_manager.cc:5484] T d3f595797050475f8e8003c3cdf6d96a P 29cb7a4d71c44e58956f1ea5afa2e2c4 reported cstate change: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New cstate: current_term: 2 leader_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.635792 7631 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f595797050475f8e8003c3cdf6d96a with delete_type TABLET_DATA_TOMBSTONED (TS 5fc8da5aaa7b4840bca58608368e0466 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48144
I20240501 19:56:09.636261 10176 tablet_replica.cc:330] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: stopping tablet replica
I20240501 19:56:09.636528 10176 raft_consensus.cc:2229] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:09.636680 10176 raft_consensus.cc:2258] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.637635 10176 ts_tablet_manager.cc:1875] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:09.639609 10176 ts_tablet_manager.cc:1888] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:09.639704 10176 log.cc:1196] T d3f595797050475f8e8003c3cdf6d96a P 5fc8da5aaa7b4840bca58608368e0466: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3f595797050475f8e8003c3cdf6d96a
I20240501 19:56:09.640298 7493 catalog_manager.cc:4830] TS 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301): tablet d3f595797050475f8e8003c3cdf6d96a (table rebalance_test_table_2 [id=e19fd899e1cb46f19ec797cf6ef227e9]) successfully deleted
I20240501 19:56:09.657182 7784 raft_consensus.cc:1261] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Refusing update from remote peer b39e25d5bdc047c3ac97e42fa43d2aa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:09.657629 9682 consensus_queue.cc:1035] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.663672 7652 raft_consensus.cc:1261] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Refusing update from remote peer 34439dd6bd754595b7219fe0a134e1eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:09.664160 10063 consensus_queue.cc:1035] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.685511 9330 consensus_queue.cc:237] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:09.686182 10023 raft_consensus.cc:2932] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } } }
I20240501 19:56:09.687559 7494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2458bf9cc6ae4ce181f9073df78ee70d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 19:56:09.688490 7504 catalog_manager.cc:5484] T 2458bf9cc6ae4ce181f9073df78ee70d P b39e25d5bdc047c3ac97e42fa43d2aa5 reported cstate change: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New cstate: current_term: 2 leader_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.690799 7631 tablet_service.cc:1514] Processing DeleteTablet for tablet 2458bf9cc6ae4ce181f9073df78ee70d with delete_type TABLET_DATA_TOMBSTONED (TS 5fc8da5aaa7b4840bca58608368e0466 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48144
I20240501 19:56:09.691093 10176 tablet_replica.cc:330] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: stopping tablet replica
I20240501 19:56:09.691226 10176 raft_consensus.cc:2229] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:09.691318 10176 raft_consensus.cc:2258] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.691965 10176 ts_tablet_manager.cc:1875] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:09.693675 10176 ts_tablet_manager.cc:1888] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:09.693739 10176 log.cc:1196] T 2458bf9cc6ae4ce181f9073df78ee70d P 5fc8da5aaa7b4840bca58608368e0466: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/2458bf9cc6ae4ce181f9073df78ee70d
I20240501 19:56:09.694185 7493 catalog_manager.cc:4830] TS 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301): tablet 2458bf9cc6ae4ce181f9073df78ee70d (table rebalance_test_table_0 [id=8ca20120f9b74a50b2a14f29cc088bf2]) successfully deleted
I20240501 19:56:09.734217 7652 raft_consensus.cc:1261] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Refusing update from remote peer 34439dd6bd754595b7219fe0a134e1eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:09.734679 10062 consensus_queue.cc:1035] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.792706 7784 raft_consensus.cc:1261] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Refusing update from remote peer 9d482feaacda4c8ab604c21f0c4b718b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:09.793140 9630 consensus_queue.cc:1035] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.824671 9330 consensus_queue.cc:237] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } }
I20240501 19:56:09.825342 10023 raft_consensus.cc:2932] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } } }
I20240501 19:56:09.827018 7494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 75f7f2b5acf54c43b00fec9c033aadc9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 19:56:09.827749 7504 catalog_manager.cc:5484] T 75f7f2b5acf54c43b00fec9c033aadc9 P b39e25d5bdc047c3ac97e42fa43d2aa5 reported cstate change: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New cstate: current_term: 2 leader_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b39e25d5bdc047c3ac97e42fa43d2aa5" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 46465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.830231 7765 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f7f2b5acf54c43b00fec9c033aadc9 with delete_type TABLET_DATA_TOMBSTONED (TS 09e60f5cde5644f78e51390fce744186 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57340
I20240501 19:56:09.830441 10173 tablet_replica.cc:330] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: stopping tablet replica
I20240501 19:56:09.830557 10173 raft_consensus.cc:2229] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:09.830662 10173 raft_consensus.cc:2258] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.831326 10173 ts_tablet_manager.cc:1875] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:09.833388 10173 ts_tablet_manager.cc:1888] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:09.833459 10173 log.cc:1196] T 75f7f2b5acf54c43b00fec9c033aadc9 P 09e60f5cde5644f78e51390fce744186: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/75f7f2b5acf54c43b00fec9c033aadc9
I20240501 19:56:09.834007 7496 catalog_manager.cc:4830] TS 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699): tablet 75f7f2b5acf54c43b00fec9c033aadc9 (table rebalance_test_table_3 [id=deba21d5b67a4c8a9b2322eca8b8faf1]) successfully deleted
I20240501 19:56:09.845785 7652 raft_consensus.cc:1261] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Refusing update from remote peer fb4dc8d240984a1c9e05c8a99afe2243: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:09.846338 10241 consensus_queue.cc:1035] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.880223 9066 consensus_queue.cc:237] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:09.880925 10063 raft_consensus.cc:2932] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } } }
I20240501 19:56:09.882414 7493 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a5b8c745ecf48a083e700b077ce10ce with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 19:56:09.883349 7509 catalog_manager.cc:5484] T 9a5b8c745ecf48a083e700b077ce10ce P 34439dd6bd754595b7219fe0a134e1eb reported cstate change: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New cstate: current_term: 2 leader_uuid: "34439dd6bd754595b7219fe0a134e1eb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.886094 7631 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5b8c745ecf48a083e700b077ce10ce with delete_type TABLET_DATA_TOMBSTONED (TS 5fc8da5aaa7b4840bca58608368e0466 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48144
I20240501 19:56:09.886307 10176 tablet_replica.cc:330] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: stopping tablet replica
I20240501 19:56:09.886457 10176 raft_consensus.cc:2229] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:09.886572 10176 raft_consensus.cc:2258] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.887285 10176 ts_tablet_manager.cc:1875] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:09.889400 10176 ts_tablet_manager.cc:1888] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:09.889474 10176 log.cc:1196] T 9a5b8c745ecf48a083e700b077ce10ce P 5fc8da5aaa7b4840bca58608368e0466: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/9a5b8c745ecf48a083e700b077ce10ce
I20240501 19:56:09.889963 7493 catalog_manager.cc:4830] TS 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301): tablet 9a5b8c745ecf48a083e700b077ce10ce (table rebalance_test_table_4 [id=886f5e1b079149088d32570e8e24019e]) successfully deleted
I20240501 19:56:09.891938 7784 raft_consensus.cc:1261] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Refusing update from remote peer 29cb7a4d71c44e58956f1ea5afa2e2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:09.892366 9645 consensus_queue.cc:1035] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.917040 8925 consensus_queue.cc:237] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } }
I20240501 19:56:09.917693 10248 raft_consensus.cc:2932] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } } }
I20240501 19:56:09.918995 7494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4846083047b478a977ae8e85e0f958d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 19:56:09.919932 7504 catalog_manager.cc:5484] T e4846083047b478a977ae8e85e0f958d P fb4dc8d240984a1c9e05c8a99afe2243 reported cstate change: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New cstate: current_term: 2 leader_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb4dc8d240984a1c9e05c8a99afe2243" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 34385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.922231 7631 tablet_service.cc:1514] Processing DeleteTablet for tablet e4846083047b478a977ae8e85e0f958d with delete_type TABLET_DATA_TOMBSTONED (TS 5fc8da5aaa7b4840bca58608368e0466 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48144
I20240501 19:56:09.922466 10176 tablet_replica.cc:330] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: stopping tablet replica
I20240501 19:56:09.922633 10176 raft_consensus.cc:2229] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:09.922731 10176 raft_consensus.cc:2258] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.923300 10176 ts_tablet_manager.cc:1875] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:09.925330 10176 ts_tablet_manager.cc:1888] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:09.925400 10176 log.cc:1196] T e4846083047b478a977ae8e85e0f958d P 5fc8da5aaa7b4840bca58608368e0466: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/e4846083047b478a977ae8e85e0f958d
I20240501 19:56:09.925882 7493 catalog_manager.cc:4830] TS 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301): tablet e4846083047b478a977ae8e85e0f958d (table rebalance_test_table_1 [id=a78b96eee0024a568693d7827bc012c8]) successfully deleted
I20240501 19:56:09.987352 9066 consensus_queue.cc:237] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } }
I20240501 19:56:09.988058 10063 raft_consensus.cc:2932] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } } }
I20240501 19:56:09.989312 7493 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b77e626fdd047dab6bbb0a271ada708 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 19:56:09.990253 7504 catalog_manager.cc:5484] T 9b77e626fdd047dab6bbb0a271ada708 P 34439dd6bd754595b7219fe0a134e1eb reported cstate change: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New cstate: current_term: 2 leader_uuid: "34439dd6bd754595b7219fe0a134e1eb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34439dd6bd754595b7219fe0a134e1eb" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 40713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.992625 7631 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b77e626fdd047dab6bbb0a271ada708 with delete_type TABLET_DATA_TOMBSTONED (TS 5fc8da5aaa7b4840bca58608368e0466 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48144
I20240501 19:56:09.992858 10176 tablet_replica.cc:330] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: stopping tablet replica
I20240501 19:56:09.992991 10176 raft_consensus.cc:2229] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:09.993098 10176 raft_consensus.cc:2258] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.993624 10176 ts_tablet_manager.cc:1875] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:09.995417 10176 ts_tablet_manager.cc:1888] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:09.995535 10176 log.cc:1196] T 9b77e626fdd047dab6bbb0a271ada708 P 5fc8da5aaa7b4840bca58608368e0466: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b77e626fdd047dab6bbb0a271ada708
I20240501 19:56:09.996078 7493 catalog_manager.cc:4830] TS 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301): tablet 9b77e626fdd047dab6bbb0a271ada708 (table rebalance_test_table_2 [id=e19fd899e1cb46f19ec797cf6ef227e9]) successfully deleted
I20240501 19:56:10.037644 8792 consensus_queue.cc:237] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } }
I20240501 19:56:10.038316 9976 raft_consensus.cc:2932] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } } }
I20240501 19:56:10.039746 7494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 19:56:10.040580 7504 catalog_manager.cc:5484] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 29cb7a4d71c44e58956f1ea5afa2e2c4 reported cstate change: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New cstate: current_term: 2 leader_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29cb7a4d71c44e58956f1ea5afa2e2c4" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 41917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:10.043092 7765 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b with delete_type TABLET_DATA_TOMBSTONED (TS 09e60f5cde5644f78e51390fce744186 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57340
I20240501 19:56:10.043340 10173 tablet_replica.cc:330] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: stopping tablet replica
I20240501 19:56:10.043486 10173 raft_consensus.cc:2229] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:10.043588 10173 raft_consensus.cc:2258] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:10.044188 10173 ts_tablet_manager.cc:1875] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:10.046094 10173 ts_tablet_manager.cc:1888] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:10.046164 10173 log.cc:1196] T 5b1a8579e6d24bdb8bee0c7fd3fd480b P 09e60f5cde5644f78e51390fce744186: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/5b1a8579e6d24bdb8bee0c7fd3fd480b
I20240501 19:56:10.046627 7496 catalog_manager.cc:4830] TS 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699): tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b (table rebalance_test_table_4 [id=886f5e1b079149088d32570e8e24019e]) successfully deleted
I20240501 19:56:10.352456 9198 consensus_queue.cc:237] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } }
I20240501 19:56:10.353091 9972 raft_consensus.cc:2932] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } } }
I20240501 19:56:10.354483 7495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb4733c007f34c4a850f9ac32a778b91 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
I20240501 19:56:10.355464 7504 catalog_manager.cc:5484] T eb4733c007f34c4a850f9ac32a778b91 P 9d482feaacda4c8ab604c21f0c4b718b reported cstate change: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New cstate: current_term: 2 leader_uuid: "9d482feaacda4c8ab604c21f0c4b718b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d482feaacda4c8ab604c21f0c4b718b" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 32985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:10.357764 7765 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4733c007f34c4a850f9ac32a778b91 with delete_type TABLET_DATA_TOMBSTONED (TS 09e60f5cde5644f78e51390fce744186 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57340
I20240501 19:56:10.357982 10173 tablet_replica.cc:330] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: stopping tablet replica
I20240501 19:56:10.358109 10173 raft_consensus.cc:2229] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:10.358214 10173 raft_consensus.cc:2258] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:10.358808 10173 ts_tablet_manager.cc:1875] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:10.360683 10173 ts_tablet_manager.cc:1888] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:10.360760 10173 log.cc:1196] T eb4733c007f34c4a850f9ac32a778b91 P 09e60f5cde5644f78e51390fce744186: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb4733c007f34c4a850f9ac32a778b91
I20240501 19:56:10.361260 7496 catalog_manager.cc:4830] TS 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699): tablet eb4733c007f34c4a850f9ac32a778b91 (table rebalance_test_table_1 [id=a78b96eee0024a568693d7827bc012c8]) successfully deleted
I20240501 19:56:10.601428 7765 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:10.603281 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.604557 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.613384 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.614933 7631 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:10.617120 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.636128 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.712473 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.712981 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.717406 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.724843 7765 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:10.726997 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.742230 7631 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:10.742902 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.964622 7765 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:10.964850 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.969085 7631 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:10.973672 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.974159 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.997418 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.002130 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.126966 7765 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:11.128285 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.132392 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.133565 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.139956 7631 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:11.143633 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.158056 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.252461 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.252461 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.252594 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.264196 7631 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:11.265753 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.268337 7765 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:11.289223 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
da61a815e04145ba9603037aebd0daa3 | 127.5.40.62:40123 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.40.20:46083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
09e60f5cde5644f78e51390fce744186 | 127.5.40.2:39699 | HEALTHY | <none> | 3 | 0
29cb7a4d71c44e58956f1ea5afa2e2c4 | 127.5.40.3:41917 | HEALTHY | <none> | 2 | 0
34439dd6bd754595b7219fe0a134e1eb | 127.5.40.5:40713 | HEALTHY | <none> | 2 | 0
5fc8da5aaa7b4840bca58608368e0466 | 127.5.40.1:43301 | HEALTHY | <none> | 3 | 0
9d482feaacda4c8ab604c21f0c4b718b | 127.5.40.6:32985 | HEALTHY | <none> | 2 | 0
b39e25d5bdc047c3ac97e42fa43d2aa5 | 127.5.40.7:46465 | HEALTHY | <none> | 2 | 0
fb4dc8d240984a1c9e05c8a99afe2243 | 127.5.40.4:34385 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.5.40.1 | experimental | 127.5.40.1:43301
local_ip_for_outbound_sockets | 127.5.40.2 | experimental | 127.5.40.2:39699
local_ip_for_outbound_sockets | 127.5.40.3 | experimental | 127.5.40.3:41917
local_ip_for_outbound_sockets | 127.5.40.4 | experimental | 127.5.40.4:34385
local_ip_for_outbound_sockets | 127.5.40.5 | experimental | 127.5.40.5:40713
local_ip_for_outbound_sockets | 127.5.40.6 | experimental | 127.5.40.6:32985
local_ip_for_outbound_sockets | 127.5.40.7 | experimental | 127.5.40.7:46465
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.40.1:43301
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.40.2:39699
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.40.3:41917
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.40.4:34385
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.40.5:40713
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.40.6:32985
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.40.7:46465
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.40.20:46083 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 19:56:11.311501 5280 log_verifier.cc:126] Checking tablet 0fead6e4699349b4a57548986c444c90
I20240501 19:56:11.322119 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0fead6e4699349b4a57548986c444c90
I20240501 19:56:11.322208 5280 log_verifier.cc:126] Checking tablet 161af226948248898c90ad36db3f0ec9
I20240501 19:56:11.330001 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 161af226948248898c90ad36db3f0ec9
I20240501 19:56:11.330071 5280 log_verifier.cc:126] Checking tablet 2458bf9cc6ae4ce181f9073df78ee70d
I20240501 19:56:11.338156 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2458bf9cc6ae4ce181f9073df78ee70d
I20240501 19:56:11.338229 5280 log_verifier.cc:126] Checking tablet 3b567cf2477443bbb17f0c7701fa1411
I20240501 19:56:11.346436 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 3b567cf2477443bbb17f0c7701fa1411
I20240501 19:56:11.346499 5280 log_verifier.cc:126] Checking tablet 408efa4d06cb40938b03d6d988a99302
I20240501 19:56:11.354212 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 408efa4d06cb40938b03d6d988a99302
I20240501 19:56:11.354279 5280 log_verifier.cc:126] Checking tablet 410cc1fb888a4d2f945c7518ee2b9580
I20240501 19:56:11.362278 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 410cc1fb888a4d2f945c7518ee2b9580
I20240501 19:56:11.362349 5280 log_verifier.cc:126] Checking tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b
I20240501 19:56:11.370762 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b
I20240501 19:56:11.370831 5280 log_verifier.cc:126] Checking tablet 6f10d321e5494c9b933877a1c74ca36d
I20240501 19:56:11.378880 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6f10d321e5494c9b933877a1c74ca36d
I20240501 19:56:11.378952 5280 log_verifier.cc:126] Checking tablet 75f7f2b5acf54c43b00fec9c033aadc9
I20240501 19:56:11.387145 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 75f7f2b5acf54c43b00fec9c033aadc9
I20240501 19:56:11.387220 5280 log_verifier.cc:126] Checking tablet 9a5b8c745ecf48a083e700b077ce10ce
I20240501 19:56:11.395313 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9a5b8c745ecf48a083e700b077ce10ce
I20240501 19:56:11.395385 5280 log_verifier.cc:126] Checking tablet 9b77e626fdd047dab6bbb0a271ada708
I20240501 19:56:11.403856 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9b77e626fdd047dab6bbb0a271ada708
I20240501 19:56:11.403924 5280 log_verifier.cc:126] Checking tablet a84bdf4685da48f18a48e8f87355fc3f
I20240501 19:56:11.410759 5280 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a84bdf4685da48f18a48e8f87355fc3f
I20240501 19:56:11.410840 5280 log_verifier.cc:126] Checking tablet ba539430c50348fc8610a2172966cb08
I20240501 19:56:11.418637 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ba539430c50348fc8610a2172966cb08
I20240501 19:56:11.418705 5280 log_verifier.cc:126] Checking tablet d3f595797050475f8e8003c3cdf6d96a
I20240501 19:56:11.426738 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d3f595797050475f8e8003c3cdf6d96a
I20240501 19:56:11.426813 5280 log_verifier.cc:126] Checking tablet e4846083047b478a977ae8e85e0f958d
I20240501 19:56:11.434870 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet e4846083047b478a977ae8e85e0f958d
I20240501 19:56:11.434934 5280 log_verifier.cc:126] Checking tablet eb4733c007f34c4a850f9ac32a778b91
I20240501 19:56:11.443166 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet eb4733c007f34c4a850f9ac32a778b91
I20240501 19:56:11.445348 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.8:0
--local_ip_for_outbound_sockets=127.5.40.8
--webserver_interface=127.5.40.8
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--builtin_ntp_servers=127.5.40.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:11.556401 10503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:11.556628 10503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:11.556670 10503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:11.559733 10503 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:11.559814 10503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:11.559885 10503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.8
I20240501 19:56:11.563704 10503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.40.8
--webserver_port=0
--tserver_master_addrs=127.5.40.62:40123
--never_fsync=true
--heap_profile_path=/tmp/kudu.10503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.8
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:11.564623 10503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:11.565724 10503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:11.574249 10509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:11.574615 10511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:11.575282 10508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:11.575852 10503 server_base.cc:1034] running on GCE node
I20240501 19:56:11.576252 10503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:11.576812 10503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:11.577960 10503 hybrid_clock.cc:648] HybridClock initialized: now 1714593371577943 us; error 34 us; skew 500 ppm
I20240501 19:56:11.579941 10503 webserver.cc:460] Webserver started at http://127.5.40.8:35995/ using document root <none> and password file <none>
I20240501 19:56:11.580499 10503 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:11.580556 10503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:11.580746 10503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:11.582477 10503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-bz40"
I20240501 19:56:11.582963 10503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-bz40"
I20240501 19:56:11.586429 10503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 19:56:11.588788 10517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:11.589846 10503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:11.589954 10503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-bz40"
I20240501 19:56:11.590056 10503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:11.606719 10503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:11.607426 10503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:11.607642 10503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:11.608204 10503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:11.609110 10503 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:11.609158 10503 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:11.609200 10503 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:11.609227 10503 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:11.617439 10503 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.8:44045
I20240501 19:56:11.617467 10630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.8:44045 every 8 connection(s)
I20240501 19:56:11.618448 10503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:11.621390 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 10503
I20240501 19:56:11.621502 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:56:11.627480 10631 heartbeater.cc:347] Connected to a master server at 127.5.40.62:40123
I20240501 19:56:11.627712 10631 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:11.628237 10631 heartbeater.cc:511] Master 127.5.40.62:40123 requested a full tablet report, sending...
I20240501 19:56:11.629221 7501 ts_manager.cc:194] Registered new tserver with Master: 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8:44045)
I20240501 19:56:11.629761 7501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.8:41973
I20240501 19:56:11.822155 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.823938 7765 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:11.825999 10565 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:11.826344 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.833832 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.841394 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.842128 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.850692 7631 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:11.958374 10565 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:11.958869 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.961223 7765 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:11.961225 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.969237 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.975747 7631 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:11.981843 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:11.992456 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:12.022658 7784 consensus_queue.cc:237] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } }
I20240501 19:56:12.023753 10725 raft_consensus.cc:2932] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } } }
I20240501 19:56:12.026526 7509 catalog_manager.cc:5484] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:12.027873 7719 consensus_peers.cc:485] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 -> Peer 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8:44045): Couldn't send request to peer 2f80f5f2062e4b7cbc146e3d2af15cbc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 408efa4d06cb40938b03d6d988a99302. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:12.034152 7652 consensus_queue.cc:237] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } }
I20240501 19:56:12.035144 10738 raft_consensus.cc:2932] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } } }
I20240501 19:56:12.038338 7509 catalog_manager.cc:5484] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:12.039417 7586 consensus_peers.cc:485] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 -> Peer 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8:44045): Couldn't send request to peer 2f80f5f2062e4b7cbc146e3d2af15cbc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 161af226948248898c90ad36db3f0ec9. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:12.530609 10771 ts_tablet_manager.cc:902] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Initiating tablet copy from peer 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:12.531318 10771 tablet_copy_client.cc:308] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:43301
I20240501 19:56:12.534794 7672 tablet_copy_service.cc:140] P 5fc8da5aaa7b4840bca58608368e0466: Received BeginTabletCopySession request for tablet 161af226948248898c90ad36db3f0ec9 from peer 2f80f5f2062e4b7cbc146e3d2af15cbc ({username='slave'} at 127.5.40.8:49085)
I20240501 19:56:12.534902 7672 tablet_copy_service.cc:161] P 5fc8da5aaa7b4840bca58608368e0466: Beginning new tablet copy session on tablet 161af226948248898c90ad36db3f0ec9 from peer 2f80f5f2062e4b7cbc146e3d2af15cbc at {username='slave'} at 127.5.40.8:49085: session id = 2f80f5f2062e4b7cbc146e3d2af15cbc-161af226948248898c90ad36db3f0ec9
I20240501 19:56:12.535624 7672 tablet_copy_source_session.cc:205] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:12.538754 10771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161af226948248898c90ad36db3f0ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:12.543964 10771 tablet_copy_client.cc:783] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:12.544164 10771 tablet_copy_client.cc:647] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:12.545805 10771 tablet_copy_client.cc:523] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:12.547597 10771 tablet_bootstrap.cc:492] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Bootstrap starting.
I20240501 19:56:12.552687 10771 log.cc:824] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:12.558954 10779 ts_tablet_manager.cc:902] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Initiating tablet copy from peer 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:12.559597 10779 tablet_copy_client.cc:308] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:39699
I20240501 19:56:12.563257 7804 tablet_copy_service.cc:140] P 09e60f5cde5644f78e51390fce744186: Received BeginTabletCopySession request for tablet 408efa4d06cb40938b03d6d988a99302 from peer 2f80f5f2062e4b7cbc146e3d2af15cbc ({username='slave'} at 127.5.40.8:33897)
I20240501 19:56:12.563375 7804 tablet_copy_service.cc:161] P 09e60f5cde5644f78e51390fce744186: Beginning new tablet copy session on tablet 408efa4d06cb40938b03d6d988a99302 from peer 2f80f5f2062e4b7cbc146e3d2af15cbc at {username='slave'} at 127.5.40.8:33897: session id = 2f80f5f2062e4b7cbc146e3d2af15cbc-408efa4d06cb40938b03d6d988a99302
I20240501 19:56:12.564118 7804 tablet_copy_source_session.cc:205] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:12.564985 10779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 408efa4d06cb40938b03d6d988a99302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:12.567286 10779 tablet_copy_client.cc:783] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:12.567586 10779 tablet_copy_client.cc:647] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:12.568794 10779 tablet_copy_client.cc:523] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:12.570065 10779 tablet_bootstrap.cc:492] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Bootstrap starting.
I20240501 19:56:12.576092 10771 tablet_bootstrap.cc:492] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:12.576390 10771 tablet_bootstrap.cc:492] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Bootstrap complete.
I20240501 19:56:12.576671 10771 ts_tablet_manager.cc:1367] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.007s
I20240501 19:56:12.579092 10771 raft_consensus.cc:348] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } }
I20240501 19:56:12.579303 10771 raft_consensus.cc:729] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f80f5f2062e4b7cbc146e3d2af15cbc, State: Initialized, Role: LEARNER
I20240501 19:56:12.579648 10771 consensus_queue.cc:260] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } }
I20240501 19:56:12.580350 10631 heartbeater.cc:503] Master 127.5.40.62:40123 was elected leader, sending a full tablet report...
I20240501 19:56:12.580520 10771 ts_tablet_manager.cc:1398] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:12.581216 7672 tablet_copy_service.cc:342] P 5fc8da5aaa7b4840bca58608368e0466: Request end of tablet copy session 2f80f5f2062e4b7cbc146e3d2af15cbc-161af226948248898c90ad36db3f0ec9 received from {username='slave'} at 127.5.40.8:49085
I20240501 19:56:12.581298 7672 tablet_copy_service.cc:433] P 5fc8da5aaa7b4840bca58608368e0466: ending tablet copy session 2f80f5f2062e4b7cbc146e3d2af15cbc-161af226948248898c90ad36db3f0ec9 on tablet 161af226948248898c90ad36db3f0ec9 with peer 2f80f5f2062e4b7cbc146e3d2af15cbc
I20240501 19:56:12.588798 10779 tablet_bootstrap.cc:492] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:12.589069 10779 tablet_bootstrap.cc:492] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Bootstrap complete.
I20240501 19:56:12.589212 10779 ts_tablet_manager.cc:1367] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240501 19:56:12.589671 10779 raft_consensus.cc:348] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } }
I20240501 19:56:12.589774 10779 raft_consensus.cc:729] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f80f5f2062e4b7cbc146e3d2af15cbc, State: Initialized, Role: LEARNER
I20240501 19:56:12.589851 10779 consensus_queue.cc:260] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: true } }
I20240501 19:56:12.590087 10779 ts_tablet_manager.cc:1398] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:12.590643 7804 tablet_copy_service.cc:342] P 09e60f5cde5644f78e51390fce744186: Request end of tablet copy session 2f80f5f2062e4b7cbc146e3d2af15cbc-408efa4d06cb40938b03d6d988a99302 received from {username='slave'} at 127.5.40.8:33897
I20240501 19:56:12.590726 7804 tablet_copy_service.cc:433] P 09e60f5cde5644f78e51390fce744186: ending tablet copy session 2f80f5f2062e4b7cbc146e3d2af15cbc-408efa4d06cb40938b03d6d988a99302 on tablet 408efa4d06cb40938b03d6d988a99302 with peer 2f80f5f2062e4b7cbc146e3d2af15cbc
I20240501 19:56:12.973806 10585 raft_consensus.cc:1203] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:13.086040 10585 raft_consensus.cc:1203] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:13.484997 10810 raft_consensus.cc:1050] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: attempting to promote NON_VOTER 2f80f5f2062e4b7cbc146e3d2af15cbc to VOTER
I20240501 19:56:13.485357 10810 consensus_queue.cc:237] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:13.486706 10585 raft_consensus.cc:1261] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 LEARNER]: Refusing update from remote peer 5fc8da5aaa7b4840bca58608368e0466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:13.487154 10809 consensus_queue.cc:1035] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:13.489674 10810 raft_consensus.cc:2932] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } } }
I20240501 19:56:13.490293 10585 raft_consensus.cc:2932] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } } }
I20240501 19:56:13.492136 7502 catalog_manager.cc:5484] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 reported cstate change: config changed from index 4 to 5, 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5fc8da5aaa7b4840bca58608368e0466" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:13.534720 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "161af226948248898c90ad36db3f0ec9"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45984
I20240501 19:56:13.534886 7652 raft_consensus.cc:595] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:13.573629 10853 raft_consensus.cc:1050] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: attempting to promote NON_VOTER 2f80f5f2062e4b7cbc146e3d2af15cbc to VOTER
I20240501 19:56:13.574016 10853 consensus_queue.cc:237] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:13.575151 10585 raft_consensus.cc:1261] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 LEARNER]: Refusing update from remote peer 09e60f5cde5644f78e51390fce744186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:13.575589 10853 consensus_queue.cc:1035] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:13.577248 10852 raft_consensus.cc:2932] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } } }
I20240501 19:56:13.577666 10585 raft_consensus.cc:2932] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } } }
I20240501 19:56:13.579849 7509 catalog_manager.cc:5484] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 reported cstate change: config changed from index 4 to 5, 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09e60f5cde5644f78e51390fce744186" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:13.752120 7652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "161af226948248898c90ad36db3f0ec9"
dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45994
I20240501 19:56:13.752265 7652 raft_consensus.cc:595] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:13.762331 7784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "408efa4d06cb40938b03d6d988a99302"
dest_uuid: "09e60f5cde5644f78e51390fce744186"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46114
I20240501 19:56:13.762475 7784 raft_consensus.cc:595] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:13.960898 10809 raft_consensus.cc:979] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: : Instructing follower 2f80f5f2062e4b7cbc146e3d2af15cbc to start an election
I20240501 19:56:13.961012 10809 raft_consensus.cc:1067] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Signalling peer 2f80f5f2062e4b7cbc146e3d2af15cbc to start an election
I20240501 19:56:13.961503 10585 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "161af226948248898c90ad36db3f0ec9"
dest_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc"
from {username='slave'} at 127.5.40.1:32859
I20240501 19:56:13.961707 10585 raft_consensus.cc:482] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:13.961817 10585 raft_consensus.cc:3037] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:13.962656 10585 raft_consensus.cc:504] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:13.963572 10585 leader_election.cc:288] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [CANDIDATE]: Term 2 election: Requested vote from peers 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301)
I20240501 19:56:13.963995 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161af226948248898c90ad36db3f0ec9" candidate_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5fc8da5aaa7b4840bca58608368e0466"
I20240501 19:56:13.964118 7652 raft_consensus.cc:3032] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:13.964165 7652 raft_consensus.cc:729] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5fc8da5aaa7b4840bca58608368e0466, State: Running, Role: LEADER
I20240501 19:56:13.964321 7652 consensus_queue.cc:260] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:13.964483 7652 raft_consensus.cc:3037] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:13.965536 7652 raft_consensus.cc:2454] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f80f5f2062e4b7cbc146e3d2af15cbc in term 2.
I20240501 19:56:13.965942 10518 leader_election.cc:302] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2f80f5f2062e4b7cbc146e3d2af15cbc, 5fc8da5aaa7b4840bca58608368e0466; no voters:
I20240501 19:56:13.966199 10785 raft_consensus.cc:2781] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:13.966372 10785 raft_consensus.cc:686] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 2 LEADER]: Becoming Leader. State: Replica: 2f80f5f2062e4b7cbc146e3d2af15cbc, State: Running, Role: LEADER
I20240501 19:56:13.966661 10785 consensus_queue.cc:237] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:13.968657 7502 catalog_manager.cc:5484] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc reported cstate change: term changed from 1 to 2, leader changed from 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) to 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8). New cstate: current_term: 2 leader_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 19:56:13.975512 7495 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 161af226948248898c90ad36db3f0ec9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8:44045) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 19:56:13.980917 7784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "408efa4d06cb40938b03d6d988a99302"
dest_uuid: "09e60f5cde5644f78e51390fce744186"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46144
I20240501 19:56:13.981040 7784 raft_consensus.cc:595] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:14.031642 10853 raft_consensus.cc:979] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: : Instructing follower 2f80f5f2062e4b7cbc146e3d2af15cbc to start an election
I20240501 19:56:14.031755 10853 raft_consensus.cc:1067] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Signalling peer 2f80f5f2062e4b7cbc146e3d2af15cbc to start an election
I20240501 19:56:14.032232 10585 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "408efa4d06cb40938b03d6d988a99302"
dest_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc"
from {username='slave'} at 127.5.40.2:50039
I20240501 19:56:14.032390 10585 raft_consensus.cc:482] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:14.032455 10585 raft_consensus.cc:3037] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:14.033262 10585 raft_consensus.cc:504] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:14.033628 10585 leader_election.cc:288] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [CANDIDATE]: Term 2 election: Requested vote from peers 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699)
I20240501 19:56:14.034044 7784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "408efa4d06cb40938b03d6d988a99302" candidate_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09e60f5cde5644f78e51390fce744186"
I20240501 19:56:14.034161 7784 raft_consensus.cc:3032] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:14.034195 7784 raft_consensus.cc:729] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09e60f5cde5644f78e51390fce744186, State: Running, Role: LEADER
I20240501 19:56:14.034317 7784 consensus_queue.cc:260] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:14.034440 7784 raft_consensus.cc:3037] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:14.035162 7784 raft_consensus.cc:2454] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f80f5f2062e4b7cbc146e3d2af15cbc in term 2.
I20240501 19:56:14.035564 10521 leader_election.cc:302] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09e60f5cde5644f78e51390fce744186, 2f80f5f2062e4b7cbc146e3d2af15cbc; no voters:
I20240501 19:56:14.035794 10785 raft_consensus.cc:2781] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:14.035890 10785 raft_consensus.cc:686] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 2 LEADER]: Becoming Leader. State: Replica: 2f80f5f2062e4b7cbc146e3d2af15cbc, State: Running, Role: LEADER
I20240501 19:56:14.036028 10785 consensus_queue.cc:237] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:14.037396 7502 catalog_manager.cc:5484] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc reported cstate change: term changed from 1 to 2, leader changed from 09e60f5cde5644f78e51390fce744186 (127.5.40.2) to 2f80f5f2062e4b7cbc146e3d2af15cbc (127.5.40.8). New cstate: current_term: 2 leader_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:14.353602 7652 raft_consensus.cc:1261] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Refusing update from remote peer 2f80f5f2062e4b7cbc146e3d2af15cbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:14.354171 10785 consensus_queue.cc:1035] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc8da5aaa7b4840bca58608368e0466" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 43301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:14.584708 10585 consensus_queue.cc:237] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:14.585433 10785 raft_consensus.cc:2932] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } } }
I20240501 19:56:14.586856 7495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 161af226948248898c90ad36db3f0ec9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 19:56:14.587853 7509 catalog_manager.cc:5484] T 161af226948248898c90ad36db3f0ec9 P 2f80f5f2062e4b7cbc146e3d2af15cbc reported cstate change: config changed from index 5 to 7, VOTER 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1) evicted. New cstate: current_term: 2 leader_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:14.590451 7631 tablet_service.cc:1514] Processing DeleteTablet for tablet 161af226948248898c90ad36db3f0ec9 with delete_type TABLET_DATA_TOMBSTONED (TS 5fc8da5aaa7b4840bca58608368e0466 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48144
I20240501 19:56:14.590768 10919 tablet_replica.cc:330] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: stopping tablet replica
I20240501 19:56:14.590883 10919 raft_consensus.cc:2229] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:14.590970 10919 raft_consensus.cc:2258] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:14.592082 10919 ts_tablet_manager.cc:1875] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:14.593840 10919 ts_tablet_manager.cc:1888] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:14.593921 10919 log.cc:1196] T 161af226948248898c90ad36db3f0ec9 P 5fc8da5aaa7b4840bca58608368e0466: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/161af226948248898c90ad36db3f0ec9
I20240501 19:56:14.594370 7493 catalog_manager.cc:4830] TS 5fc8da5aaa7b4840bca58608368e0466 (127.5.40.1:43301): tablet 161af226948248898c90ad36db3f0ec9 (table rebalance_test_table_3 [id=deba21d5b67a4c8a9b2322eca8b8faf1]) successfully deleted
I20240501 19:56:14.650141 7784 raft_consensus.cc:1261] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Refusing update from remote peer 2f80f5f2062e4b7cbc146e3d2af15cbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:14.650604 10785 consensus_queue.cc:1035] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e60f5cde5644f78e51390fce744186" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 39699 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:15.183483 10585 consensus_queue.cc:237] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } }
I20240501 19:56:15.184188 10881 raft_consensus.cc:2932] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } } }
I20240501 19:56:15.185549 7495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 408efa4d06cb40938b03d6d988a99302 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
I20240501 19:56:15.186489 7509 catalog_manager.cc:5484] T 408efa4d06cb40938b03d6d988a99302 P 2f80f5f2062e4b7cbc146e3d2af15cbc reported cstate change: config changed from index 5 to 7, VOTER 09e60f5cde5644f78e51390fce744186 (127.5.40.2) evicted. New cstate: current_term: 2 leader_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f80f5f2062e4b7cbc146e3d2af15cbc" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:15.189249 7765 tablet_service.cc:1514] Processing DeleteTablet for tablet 408efa4d06cb40938b03d6d988a99302 with delete_type TABLET_DATA_TOMBSTONED (TS 09e60f5cde5644f78e51390fce744186 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57340
I20240501 19:56:15.189582 10953 tablet_replica.cc:330] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: stopping tablet replica
I20240501 19:56:15.189699 10953 raft_consensus.cc:2229] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:15.189788 10953 raft_consensus.cc:2258] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:15.190902 10953 ts_tablet_manager.cc:1875] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:15.192770 10953 ts_tablet_manager.cc:1888] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:15.192836 10953 log.cc:1196] T 408efa4d06cb40938b03d6d988a99302 P 09e60f5cde5644f78e51390fce744186: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/408efa4d06cb40938b03d6d988a99302
I20240501 19:56:15.193351 7496 catalog_manager.cc:4830] TS 09e60f5cde5644f78e51390fce744186 (127.5.40.2:39699): tablet 408efa4d06cb40938b03d6d988a99302 (table rebalance_test_table_2 [id=e19fd899e1cb46f19ec797cf6ef227e9]) successfully deleted
I20240501 19:56:15.305202 7765 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.307091 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.307327 10565 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.309583 7631 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.310704 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.321069 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.324184 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.333778 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.415294 10565 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.415524 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.416234 7765 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.425602 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.434403 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.438378 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.440339 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.445273 7631 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.532383 10565 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.534292 8772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.534727 7765 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.537828 9178 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.552675 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.557680 9046 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.562108 9310 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:15.571775 7631 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
da61a815e04145ba9603037aebd0daa3 | 127.5.40.62:40123 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.40.20:46083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
09e60f5cde5644f78e51390fce744186 | 127.5.40.2:39699 | HEALTHY | <none> | 2 | 0
29cb7a4d71c44e58956f1ea5afa2e2c4 | 127.5.40.3:41917 | HEALTHY | <none> | 2 | 0
2f80f5f2062e4b7cbc146e3d2af15cbc | 127.5.40.8:44045 | HEALTHY | <none> | 2 | 0
34439dd6bd754595b7219fe0a134e1eb | 127.5.40.5:40713 | HEALTHY | <none> | 2 | 0
5fc8da5aaa7b4840bca58608368e0466 | 127.5.40.1:43301 | HEALTHY | <none> | 2 | 0
9d482feaacda4c8ab604c21f0c4b718b | 127.5.40.6:32985 | HEALTHY | <none> | 2 | 0
b39e25d5bdc047c3ac97e42fa43d2aa5 | 127.5.40.7:46465 | HEALTHY | <none> | 2 | 0
fb4dc8d240984a1c9e05c8a99afe2243 | 127.5.40.4:34385 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.5.40.1 | experimental | 127.5.40.1:43301
local_ip_for_outbound_sockets | 127.5.40.2 | experimental | 127.5.40.2:39699
local_ip_for_outbound_sockets | 127.5.40.3 | experimental | 127.5.40.3:41917
local_ip_for_outbound_sockets | 127.5.40.4 | experimental | 127.5.40.4:34385
local_ip_for_outbound_sockets | 127.5.40.5 | experimental | 127.5.40.5:40713
local_ip_for_outbound_sockets | 127.5.40.6 | experimental | 127.5.40.6:32985
local_ip_for_outbound_sockets | 127.5.40.7 | experimental | 127.5.40.7:46465
local_ip_for_outbound_sockets | 127.5.40.8 | experimental | 127.5.40.8:44045
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.40.1:43301
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.40.2:39699
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.40.3:41917
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.40.4:34385
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.40.5:40713
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.40.6:32985
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.40.7:46465
server_dump_info_path | /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.5.40.8:44045
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.40.20:46083 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 19:56:15.593384 5280 log_verifier.cc:126] Checking tablet 0fead6e4699349b4a57548986c444c90
I20240501 19:56:15.600406 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0fead6e4699349b4a57548986c444c90
I20240501 19:56:15.600488 5280 log_verifier.cc:126] Checking tablet 161af226948248898c90ad36db3f0ec9
I20240501 19:56:15.609541 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 161af226948248898c90ad36db3f0ec9
I20240501 19:56:15.609611 5280 log_verifier.cc:126] Checking tablet 2458bf9cc6ae4ce181f9073df78ee70d
I20240501 19:56:15.615113 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2458bf9cc6ae4ce181f9073df78ee70d
I20240501 19:56:15.615170 5280 log_verifier.cc:126] Checking tablet 3b567cf2477443bbb17f0c7701fa1411
I20240501 19:56:15.620718 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 3b567cf2477443bbb17f0c7701fa1411
I20240501 19:56:15.620777 5280 log_verifier.cc:126] Checking tablet 408efa4d06cb40938b03d6d988a99302
I20240501 19:56:15.628826 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 408efa4d06cb40938b03d6d988a99302
I20240501 19:56:15.628888 5280 log_verifier.cc:126] Checking tablet 410cc1fb888a4d2f945c7518ee2b9580
I20240501 19:56:15.634137 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 410cc1fb888a4d2f945c7518ee2b9580
I20240501 19:56:15.634191 5280 log_verifier.cc:126] Checking tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b
I20240501 19:56:15.639851 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5b1a8579e6d24bdb8bee0c7fd3fd480b
I20240501 19:56:15.639900 5280 log_verifier.cc:126] Checking tablet 6f10d321e5494c9b933877a1c74ca36d
I20240501 19:56:15.645345 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6f10d321e5494c9b933877a1c74ca36d
I20240501 19:56:15.645392 5280 log_verifier.cc:126] Checking tablet 75f7f2b5acf54c43b00fec9c033aadc9
I20240501 19:56:15.650889 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 75f7f2b5acf54c43b00fec9c033aadc9
I20240501 19:56:15.650941 5280 log_verifier.cc:126] Checking tablet 9a5b8c745ecf48a083e700b077ce10ce
I20240501 19:56:15.656801 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9a5b8c745ecf48a083e700b077ce10ce
I20240501 19:56:15.656850 5280 log_verifier.cc:126] Checking tablet 9b77e626fdd047dab6bbb0a271ada708
I20240501 19:56:15.662418 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9b77e626fdd047dab6bbb0a271ada708
I20240501 19:56:15.662467 5280 log_verifier.cc:126] Checking tablet a84bdf4685da48f18a48e8f87355fc3f
I20240501 19:56:15.666949 5280 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a84bdf4685da48f18a48e8f87355fc3f
I20240501 19:56:15.666997 5280 log_verifier.cc:126] Checking tablet ba539430c50348fc8610a2172966cb08
I20240501 19:56:15.672369 5280 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ba539430c50348fc8610a2172966cb08
I20240501 19:56:15.672421 5280 log_verifier.cc:126] Checking tablet d3f595797050475f8e8003c3cdf6d96a
I20240501 19:56:15.678026 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d3f595797050475f8e8003c3cdf6d96a
I20240501 19:56:15.678076 5280 log_verifier.cc:126] Checking tablet e4846083047b478a977ae8e85e0f958d
I20240501 19:56:15.683689 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet e4846083047b478a977ae8e85e0f958d
I20240501 19:56:15.683737 5280 log_verifier.cc:126] Checking tablet eb4733c007f34c4a850f9ac32a778b91
I20240501 19:56:15.689395 5280 log_verifier.cc:177] Verified matching terms for 7 ops in tablet eb4733c007f34c4a850f9ac32a778b91
I20240501 19:56:15.689894 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 7569
I20240501 19:56:15.695906 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 7702
I20240501 19:56:15.701436 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 8709
I20240501 19:56:15.706612 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 8841
I20240501 19:56:15.711185 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 8976
I20240501 19:56:15.715797 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 9115
I20240501 19:56:15.720803 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 9247
I20240501 19:56:15.725282 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 10503
I20240501 19:56:15.729702 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 7478
2024-05-01T19:56:15Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (16576 ms)
[----------] 1 test from RebalanceParamTest (16576 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240501 19:56:15.762811 5280 test_util.cc:268] Using random seed: -518473356
I20240501 19:56:15.763643 5280 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:56:15.763686 5280 ts_itest-base.cc:113] --------------
I20240501 19:56:15.763696 5280 ts_itest-base.cc:114] 8 tablet servers
I20240501 19:56:15.763705 5280 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:56:15.763713 5280 ts_itest-base.cc:116] --------------
2024-05-01T19:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:15Z Disabled control of system clock
I20240501 19:56:15.771863 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.40.62:42037
--webserver_interface=127.5.40.62
--webserver_port=0
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.40.62:42037
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 19:56:15.879961 11074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:15.880219 11074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:15.880277 11074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:15.883522 11074 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:15.883605 11074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:15.883637 11074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:15.883671 11074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:15.883708 11074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:15.887521 11074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.40.62:42037
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.40.62:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.40.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.11074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:15.888464 11074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:15.889633 11074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:15.897095 11082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.897665 11080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.898072 11079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:15.898667 11074 server_base.cc:1034] running on GCE node
I20240501 19:56:15.899191 11074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:15.900171 11074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:15.901316 11074 hybrid_clock.cc:648] HybridClock initialized: now 1714593375901304 us; error 28 us; skew 500 ppm
I20240501 19:56:15.903183 11074 webserver.cc:460] Webserver started at http://127.5.40.62:33407/ using document root <none> and password file <none>
I20240501 19:56:15.903748 11074 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:15.903803 11074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:15.903992 11074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:15.905652 11074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "351e1e4f69a849a0b27128be4e72e5bb"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-bz40"
I20240501 19:56:15.906128 11074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "351e1e4f69a849a0b27128be4e72e5bb"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-bz40"
I20240501 19:56:15.909374 11074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:15.911448 11088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.912415 11074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:15.912530 11074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
uuid: "351e1e4f69a849a0b27128be4e72e5bb"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-bz40"
I20240501 19:56:15.912608 11074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:15.941593 11074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:15.942337 11074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:15.942516 11074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:15.949280 11074 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.62:42037
I20240501 19:56:15.949286 11141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.62:42037 every 8 connection(s)
I20240501 19:56:15.950280 11074 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:56:15.952888 11142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:15.956745 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 11074
I20240501 19:56:15.956900 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:56:15.958412 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Bootstrap starting.
I20240501 19:56:15.960633 11142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:15.961328 11142 log.cc:824] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:15.962966 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: No bootstrap required, opened a new log
I20240501 19:56:15.965886 11142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351e1e4f69a849a0b27128be4e72e5bb" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 42037 } }
I20240501 19:56:15.966099 11142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:15.966158 11142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351e1e4f69a849a0b27128be4e72e5bb, State: Initialized, Role: FOLLOWER
I20240501 19:56:15.966567 11142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351e1e4f69a849a0b27128be4e72e5bb" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 42037 } }
I20240501 19:56:15.966688 11142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:15.966744 11142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:15.966842 11142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:15.967583 11142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351e1e4f69a849a0b27128be4e72e5bb" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 42037 } }
I20240501 19:56:15.967886 11142 leader_election.cc:302] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 351e1e4f69a849a0b27128be4e72e5bb; no voters:
I20240501 19:56:15.968142 11142 leader_election.cc:288] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:15.968500 11147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:15.968689 11147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [term 1 LEADER]: Becoming Leader. State: Replica: 351e1e4f69a849a0b27128be4e72e5bb, State: Running, Role: LEADER
I20240501 19:56:15.968946 11147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351e1e4f69a849a0b27128be4e72e5bb" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 42037 } }
I20240501 19:56:15.969163 11142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:15.970571 11149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 351e1e4f69a849a0b27128be4e72e5bb. Latest consensus state: current_term: 1 leader_uuid: "351e1e4f69a849a0b27128be4e72e5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351e1e4f69a849a0b27128be4e72e5bb" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 42037 } } }
I20240501 19:56:15.970571 11148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "351e1e4f69a849a0b27128be4e72e5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351e1e4f69a849a0b27128be4e72e5bb" member_type: VOTER last_known_addr { host: "127.5.40.62" port: 42037 } } }
I20240501 19:56:15.970690 11148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:15.970690 11149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:15.971077 11160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:15.973831 11160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:15.978482 11160 catalog_manager.cc:1347] Generated new cluster ID: 19a1d2c6ad534748852753baee6281c6
I20240501 19:56:15.978560 11160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:15.999284 11160 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:16.000228 11160 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:16.009299 11160 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Generated new TSK 0
I20240501 19:56:16.010178 11160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:16.019672 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.1:0
--local_ip_for_outbound_sockets=127.5.40.1
--webserver_interface=127.5.40.1
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:16.126539 11166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:16.126757 11166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:16.126806 11166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:16.129863 11166 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:16.129930 11166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:16.130029 11166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.1
I20240501 19:56:16.133592 11166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.40.1
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.11166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.1
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:16.134470 11166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:16.135593 11166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:16.143584 11172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.144119 11174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.144491 11171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:16.145195 11166 server_base.cc:1034] running on GCE node
I20240501 19:56:16.145567 11166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:16.146121 11166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:16.147270 11166 hybrid_clock.cc:648] HybridClock initialized: now 1714593376147251 us; error 32 us; skew 500 ppm
I20240501 19:56:16.149149 11166 webserver.cc:460] Webserver started at http://127.5.40.1:41591/ using document root <none> and password file <none>
I20240501 19:56:16.149691 11166 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:16.149747 11166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:16.149919 11166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:16.151585 11166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.152058 11166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.155333 11166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:16.157493 11180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.158499 11166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:16.158610 11166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.158694 11166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:16.170550 11166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:16.171200 11166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:16.171356 11166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:16.171891 11166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:16.172794 11166 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:16.172849 11166 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.172904 11166 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:16.172930 11166 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.180980 11166 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.1:42965
I20240501 19:56:16.181003 11294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.1:42965 every 8 connection(s)
I20240501 19:56:16.181963 11166 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:56:16.185504 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 11166
I20240501 19:56:16.185606 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:56:16.187867 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.2:0
--local_ip_for_outbound_sockets=127.5.40.2
--webserver_interface=127.5.40.2
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:16.191108 11295 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:16.191372 11295 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:16.192081 11295 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:16.193593 11106 ts_manager.cc:194] Registered new tserver with Master: 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:16.194778 11106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.1:59429
W20240501 19:56:16.294031 11298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:16.294270 11298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:16.294328 11298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:16.297446 11298 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:16.297518 11298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:16.297606 11298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.2
I20240501 19:56:16.301165 11298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.40.2
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.11298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.2
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:16.302081 11298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:16.303167 11298 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:16.311367 11304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.311728 11306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.312258 11303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:16.312865 11298 server_base.cc:1034] running on GCE node
I20240501 19:56:16.313225 11298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:16.313769 11298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:16.314909 11298 hybrid_clock.cc:648] HybridClock initialized: now 1714593376314897 us; error 28 us; skew 500 ppm
I20240501 19:56:16.316802 11298 webserver.cc:460] Webserver started at http://127.5.40.2:35805/ using document root <none> and password file <none>
I20240501 19:56:16.317337 11298 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:16.317392 11298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:16.317564 11298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:16.319168 11298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ef60ab93b49c4346af4f62d3b98558b8"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.319630 11298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ef60ab93b49c4346af4f62d3b98558b8"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.322744 11298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:16.324775 11312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.325695 11298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:16.325793 11298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ef60ab93b49c4346af4f62d3b98558b8"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.325881 11298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:16.341286 11298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:16.341858 11298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:16.342005 11298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:16.342497 11298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:16.343353 11298 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:16.343398 11298 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.343489 11298 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:16.343515 11298 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.351477 11298 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.2:40185
I20240501 19:56:16.351506 11426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.2:40185 every 8 connection(s)
I20240501 19:56:16.352424 11298 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:56:16.353296 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 11298
I20240501 19:56:16.353410 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:56:16.355636 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.3:0
--local_ip_for_outbound_sockets=127.5.40.3
--webserver_interface=127.5.40.3
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:16.361119 11427 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:16.361369 11427 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:16.362118 11427 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:16.363235 11106 ts_manager.cc:194] Registered new tserver with Master: ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:16.363811 11106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.2:49015
W20240501 19:56:16.461164 11430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:16.461369 11430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:16.461414 11430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:16.464552 11430 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:16.464622 11430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:16.464697 11430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.3
I20240501 19:56:16.468286 11430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.40.3
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.11430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.3
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:16.469183 11430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:16.470336 11430 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:16.478327 11438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.478888 11436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.479370 11435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:16.480012 11430 server_base.cc:1034] running on GCE node
I20240501 19:56:16.480428 11430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:16.480968 11430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:16.482108 11430 hybrid_clock.cc:648] HybridClock initialized: now 1714593376482093 us; error 31 us; skew 500 ppm
I20240501 19:56:16.483892 11430 webserver.cc:460] Webserver started at http://127.5.40.3:43147/ using document root <none> and password file <none>
I20240501 19:56:16.484421 11430 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:16.484478 11430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:16.484639 11430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:16.486214 11430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "08711534813d47ba85aaec0771ed5d0f"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.486657 11430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "08711534813d47ba85aaec0771ed5d0f"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.489800 11430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 19:56:16.491883 11444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.492877 11430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:16.492988 11430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "08711534813d47ba85aaec0771ed5d0f"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.493070 11430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:16.511101 11430 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:16.511811 11430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:16.511986 11430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:16.512527 11430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:16.513427 11430 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:16.513475 11430 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.513530 11430 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:16.513553 11430 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.521433 11430 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.3:45965
I20240501 19:56:16.521457 11558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.3:45965 every 8 connection(s)
I20240501 19:56:16.522383 11430 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:56:16.530519 11559 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:16.530731 11559 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:16.531082 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 11430
I20240501 19:56:16.531172 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:56:16.531277 11559 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:16.532258 11105 ts_manager.cc:194] Registered new tserver with Master: 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:16.532912 11105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.3:39499
I20240501 19:56:16.533385 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.4:0
--local_ip_for_outbound_sockets=127.5.40.4
--webserver_interface=127.5.40.4
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:16.637383 11562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:16.637603 11562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:16.637650 11562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:16.640619 11562 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:16.640686 11562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:16.640779 11562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.4
I20240501 19:56:16.644325 11562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.40.4
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.11562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.4
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:16.645220 11562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:16.646311 11562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:16.654690 11567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.655066 11568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.655561 11570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:16.656062 11562 server_base.cc:1034] running on GCE node
I20240501 19:56:16.656457 11562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:16.657024 11562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:16.658169 11562 hybrid_clock.cc:648] HybridClock initialized: now 1714593376658149 us; error 32 us; skew 500 ppm
I20240501 19:56:16.660105 11562 webserver.cc:460] Webserver started at http://127.5.40.4:41349/ using document root <none> and password file <none>
I20240501 19:56:16.660645 11562 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:16.660703 11562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:16.660872 11562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:16.662585 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8ad18cd54c6340fd92b874157875fa49"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.663107 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8ad18cd54c6340fd92b874157875fa49"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.666419 11562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 19:56:16.668530 11576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.669483 11562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:16.669584 11562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8ad18cd54c6340fd92b874157875fa49"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.669703 11562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:16.688480 11562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:16.689158 11562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:16.689330 11562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:16.689891 11562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:16.690809 11562 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:16.690858 11562 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.690912 11562 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:16.690940 11562 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.699147 11562 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.4:42041
I20240501 19:56:16.699214 11689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.4:42041 every 8 connection(s)
I20240501 19:56:16.700145 11562 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:16.708338 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 11562
I20240501 19:56:16.708451 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:56:16.708534 11690 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:16.708771 11690 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:16.709389 11690 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:16.710410 11105 ts_manager.cc:194] Registered new tserver with Master: 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:16.710906 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.5:0
--local_ip_for_outbound_sockets=127.5.40.5
--webserver_interface=127.5.40.5
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:16.711103 11105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.4:59009
W20240501 19:56:16.817296 11693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:16.817518 11693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:16.817565 11693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:16.820530 11693 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:16.820593 11693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:16.820674 11693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.5
I20240501 19:56:16.824244 11693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.40.5
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.11693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.5
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:16.825156 11693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:16.826277 11693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:16.834399 11701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.834790 11698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.835381 11699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:16.835928 11693 server_base.cc:1034] running on GCE node
I20240501 19:56:16.836302 11693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:16.836875 11693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:16.838016 11693 hybrid_clock.cc:648] HybridClock initialized: now 1714593376838002 us; error 28 us; skew 500 ppm
I20240501 19:56:16.839876 11693 webserver.cc:460] Webserver started at http://127.5.40.5:36289/ using document root <none> and password file <none>
I20240501 19:56:16.840413 11693 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:16.840471 11693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:16.840638 11693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:16.842279 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.842730 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.846021 11693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 19:56:16.848110 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.849083 11693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:16.849189 11693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:16.849270 11693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:16.868209 11693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:16.868799 11693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:16.868959 11693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:16.869534 11693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:16.870579 11693 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:16.870668 11693 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.870734 11693 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:16.870755 11693 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.879240 11693 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.5:42501
I20240501 19:56:16.879257 11820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.5:42501 every 8 connection(s)
I20240501 19:56:16.880190 11693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:16.886368 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 11693
I20240501 19:56:16.886477 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:56:16.888669 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.6:0
--local_ip_for_outbound_sockets=127.5.40.6
--webserver_interface=127.5.40.6
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:16.888672 11821 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:16.888880 11821 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:16.889395 11821 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:16.890384 11105 ts_manager.cc:194] Registered new tserver with Master: 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:16.890924 11105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.5:59997
W20240501 19:56:16.995123 11824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:16.995335 11824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:16.995400 11824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:16.998349 11824 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:16.998411 11824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:16.998492 11824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.6
I20240501 19:56:17.001941 11824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.40.6
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.11824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.6
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:17.002923 11824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:17.004016 11824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:17.011992 11830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.012388 11829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.013094 11832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:17.013602 11824 server_base.cc:1034] running on GCE node
I20240501 19:56:17.013933 11824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:17.014446 11824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:17.015611 11824 hybrid_clock.cc:648] HybridClock initialized: now 1714593377015592 us; error 29 us; skew 500 ppm
I20240501 19:56:17.017446 11824 webserver.cc:460] Webserver started at http://127.5.40.6:45411/ using document root <none> and password file <none>
I20240501 19:56:17.017973 11824 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:17.018031 11824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:17.018195 11824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:17.019850 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:17.020292 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:17.023425 11824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 19:56:17.025470 11838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.026420 11824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:17.026520 11824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:17.026597 11824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:17.056829 11824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:17.057493 11824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:17.057651 11824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:17.058213 11824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:17.059095 11824 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:17.059144 11824 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.059199 11824 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:17.059227 11824 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.068645 11824 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.6:46331
I20240501 19:56:17.068683 11951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.6:46331 every 8 connection(s)
I20240501 19:56:17.069830 11824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:17.074103 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 11824
I20240501 19:56:17.074234 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:56:17.076447 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.7:0
--local_ip_for_outbound_sockets=127.5.40.7
--webserver_interface=127.5.40.7
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:17.079331 11952 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:17.079571 11952 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:17.080091 11952 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:17.081089 11105 ts_manager.cc:194] Registered new tserver with Master: 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:17.081640 11105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.6:50871
W20240501 19:56:17.181360 11955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:17.181581 11955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:17.181624 11955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:17.184573 11955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:17.184634 11955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:17.184707 11955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.7
I20240501 19:56:17.188163 11955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.40.7
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.11955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.7
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:17.188983 11955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:17.190040 11955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:56:17.197486 11295 heartbeater.cc:503] Master 127.5.40.62:42037 was elected leader, sending a full tablet report...
W20240501 19:56:17.198215 11961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.198786 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.199060 11960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:17.199800 11955 server_base.cc:1034] running on GCE node
I20240501 19:56:17.200192 11955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:17.200721 11955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:17.201879 11955 hybrid_clock.cc:648] HybridClock initialized: now 1714593377201860 us; error 41 us; skew 500 ppm
I20240501 19:56:17.203652 11955 webserver.cc:460] Webserver started at http://127.5.40.7:35033/ using document root <none> and password file <none>
I20240501 19:56:17.204182 11955 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:17.204243 11955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:17.204409 11955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:17.205968 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6921cf2e41f64d66842885512da8de6c"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:17.206401 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6921cf2e41f64d66842885512da8de6c"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:17.209564 11955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 19:56:17.211696 11969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.212675 11955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:17.212772 11955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6921cf2e41f64d66842885512da8de6c"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:17.212854 11955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:17.239778 11955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:17.240440 11955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:17.240599 11955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:17.241135 11955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:17.242064 11955 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:17.242110 11955 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.242163 11955 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:17.242189 11955 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.252243 11955 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.7:36243
I20240501 19:56:17.252271 12082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.7:36243 every 8 connection(s)
I20240501 19:56:17.253233 11955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:17.261662 12083 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:17.261885 12083 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:17.262475 12083 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:17.262485 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 11955
I20240501 19:56:17.262581 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:56:17.263698 11106 ts_manager.cc:194] Registered new tserver with Master: 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:17.264236 11106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.7:55647
I20240501 19:56:17.265729 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.8:0
--local_ip_for_outbound_sockets=127.5.40.8
--webserver_interface=127.5.40.8
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:17.366158 11427 heartbeater.cc:503] Master 127.5.40.62:42037 was elected leader, sending a full tablet report...
W20240501 19:56:17.373286 12086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:17.373505 12086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:17.373549 12086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:17.376591 12086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:17.376657 12086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:17.376741 12086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.8
I20240501 19:56:17.380291 12086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.40.8
--webserver_port=0
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.12086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.8
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:17.381166 12086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:17.382301 12086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:17.390220 12091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.390781 12092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.391211 12094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:17.391917 12086 server_base.cc:1034] running on GCE node
I20240501 19:56:17.392282 12086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:17.392807 12086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:17.393939 12086 hybrid_clock.cc:648] HybridClock initialized: now 1714593377393924 us; error 38 us; skew 500 ppm
I20240501 19:56:17.395788 12086 webserver.cc:460] Webserver started at http://127.5.40.8:43041/ using document root <none> and password file <none>
I20240501 19:56:17.396317 12086 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:17.396368 12086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:17.396508 12086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:17.398144 12086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "533908e517b24f77b2af8b5863e4ae5a"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:17.398569 12086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "533908e517b24f77b2af8b5863e4ae5a"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:17.401798 12086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 19:56:17.403797 12100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.404755 12086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:17.404850 12086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "533908e517b24f77b2af8b5863e4ae5a"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:17.404932 12086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:17.420008 12086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:17.420601 12086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:17.420758 12086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:17.421303 12086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:17.422190 12086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:17.422237 12086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.422281 12086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:17.422309 12086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.430433 12086 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.8:34789
I20240501 19:56:17.430450 12213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.8:34789 every 8 connection(s)
I20240501 19:56:17.431421 12086 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:17.439965 12214 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:17.440203 12214 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:17.440757 12214 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:17.441233 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 12086
I20240501 19:56:17.441351 5280 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:56:17.441792 11105 ts_manager.cc:194] Registered new tserver with Master: 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:17.442355 11105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.8:38591
I20240501 19:56:17.443630 5280 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:56:17.453109 5280 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:56:17.453188 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 11693
I20240501 19:56:17.458036 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 11824
I20240501 19:56:17.462219 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 11955
I20240501 19:56:17.466435 5280 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu with pid 12086
I20240501 19:56:17.534929 11559 heartbeater.cc:503] Master 127.5.40.62:42037 was elected leader, sending a full tablet report...
I20240501 19:56:17.713315 11690 heartbeater.cc:503] Master 127.5.40.62:42037 was elected leader, sending a full tablet report...
I20240501 19:56:21.222097 11105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:21.242060 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 171bb2e007d1427ebf737bb9fa8b75a8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.242091 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 171bb2e007d1427ebf737bb9fa8b75a8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.242311 11492 tablet_service.cc:1467] Processing CreateTablet for tablet b3a53d7da1204a2a9cb0869330285284 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.242091 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd620de3ef44eab9cd021874be0a3e4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.243011 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171bb2e007d1427ebf737bb9fa8b75a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.243011 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a53d7da1204a2a9cb0869330285284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.243012 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171bb2e007d1427ebf737bb9fa8b75a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.243327 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd620de3ef44eab9cd021874be0a3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.242060 11359 tablet_service.cc:1467] Processing CreateTablet for tablet b3a53d7da1204a2a9cb0869330285284 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.244592 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a53d7da1204a2a9cb0869330285284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.245906 11228 tablet_service.cc:1467] Processing CreateTablet for tablet b3a53d7da1204a2a9cb0869330285284 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.246135 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a53d7da1204a2a9cb0869330285284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.247859 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd620de3ef44eab9cd021874be0a3e4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.248085 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd620de3ef44eab9cd021874be0a3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.249471 12234 tablet_bootstrap.cc:492] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:21.250219 12235 tablet_bootstrap.cc:492] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:21.250916 12236 tablet_bootstrap.cc:492] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:21.251538 12234 tablet_bootstrap.cc:654] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.252869 12235 tablet_bootstrap.cc:654] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.254412 12234 log.cc:824] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:21.254835 12235 log.cc:824] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:21.255313 12236 tablet_bootstrap.cc:654] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.256263 12235 tablet_bootstrap.cc:492] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:21.256263 12234 tablet_bootstrap.cc:492] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:21.256433 12235 ts_tablet_manager.cc:1367] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:56:21.256433 12234 ts_tablet_manager.cc:1367] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240501 19:56:21.256608 12236 log.cc:824] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:21.260030 12235 raft_consensus.cc:348] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.260228 12235 raft_consensus.cc:374] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.260273 12235 raft_consensus.cc:729] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.260315 12234 raft_consensus.cc:348] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.260471 12234 raft_consensus.cc:374] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.260520 12234 raft_consensus.cc:729] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.260730 12235 consensus_queue.cc:260] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.260843 12236 tablet_bootstrap.cc:492] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:21.260888 12234 consensus_queue.cc:260] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.260996 12236 ts_tablet_manager.cc:1367] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 19:56:21.260663 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd620de3ef44eab9cd021874be0a3e4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.261581 12234 ts_tablet_manager.cc:1398] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240501 19:56:21.261581 12235 ts_tablet_manager.cc:1398] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240501 19:56:21.261636 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd620de3ef44eab9cd021874be0a3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.261912 12234 tablet_bootstrap.cc:492] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:21.261910 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 171bb2e007d1427ebf737bb9fa8b75a8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.262912 12235 tablet_bootstrap.cc:492] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:21.262930 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171bb2e007d1427ebf737bb9fa8b75a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.263854 12235 tablet_bootstrap.cc:654] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.264708 12236 raft_consensus.cc:348] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.264886 12235 tablet_bootstrap.cc:492] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:21.264966 12235 ts_tablet_manager.cc:1367] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:21.264987 12236 raft_consensus.cc:374] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.265121 12236 raft_consensus.cc:729] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.265477 12235 raft_consensus.cc:348] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.265587 12235 raft_consensus.cc:374] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.265625 12235 raft_consensus.cc:729] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.265690 12236 consensus_queue.cc:260] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.265753 12235 consensus_queue.cc:260] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.266136 12235 ts_tablet_manager.cc:1398] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:21.266628 12236 ts_tablet_manager.cc:1398] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240501 19:56:21.266866 12234 tablet_bootstrap.cc:654] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.267042 12236 tablet_bootstrap.cc:492] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:21.267961 12236 tablet_bootstrap.cc:654] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.268628 12234 tablet_bootstrap.cc:492] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:21.268710 12234 ts_tablet_manager.cc:1367] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240501 19:56:21.269230 12234 raft_consensus.cc:348] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.269315 12234 raft_consensus.cc:374] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.269343 12234 raft_consensus.cc:729] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.269460 12234 consensus_queue.cc:260] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.269762 12234 ts_tablet_manager.cc:1398] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:21.270685 12236 tablet_bootstrap.cc:492] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:21.270859 12236 ts_tablet_manager.cc:1367] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:21.272065 12236 raft_consensus.cc:348] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.272217 12236 raft_consensus.cc:374] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.272264 12236 raft_consensus.cc:729] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.272383 12236 consensus_queue.cc:260] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.272641 12236 ts_tablet_manager.cc:1398] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:21.272802 12236 tablet_bootstrap.cc:492] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:21.273363 12247 tablet_bootstrap.cc:492] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:21.274333 12236 tablet_bootstrap.cc:654] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.275431 12247 tablet_bootstrap.cc:654] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.275995 12236 tablet_bootstrap.cc:492] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:21.276079 12236 ts_tablet_manager.cc:1367] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:21.276280 12247 log.cc:824] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: Log is configured to *not* fsync() on all Append() calls
W20240501 19:56:21.276286 11560 tablet.cc:2369] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:21.276499 12236 raft_consensus.cc:348] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.276602 12236 raft_consensus.cc:374] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.276639 12236 raft_consensus.cc:729] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.276757 12236 consensus_queue.cc:260] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.277001 12236 ts_tablet_manager.cc:1398] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:21.277812 12247 tablet_bootstrap.cc:492] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:21.277968 12247 ts_tablet_manager.cc:1367] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:21.280817 12247 raft_consensus.cc:348] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.280992 12247 raft_consensus.cc:374] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.281054 12247 raft_consensus.cc:729] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.281442 12247 consensus_queue.cc:260] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.282254 12247 ts_tablet_manager.cc:1398] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:21.282491 12247 tablet_bootstrap.cc:492] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:21.283344 12247 tablet_bootstrap.cc:654] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.284679 12247 tablet_bootstrap.cc:492] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:21.284778 12247 ts_tablet_manager.cc:1367] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:21.285313 12247 raft_consensus.cc:348] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.285420 12247 raft_consensus.cc:374] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.285442 12247 raft_consensus.cc:729] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.285533 12247 consensus_queue.cc:260] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.285748 12247 ts_tablet_manager.cc:1398] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:21.306130 12241 raft_consensus.cc:482] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.306344 12241 raft_consensus.cc:504] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.307557 12241 leader_election.cc:288] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:21.311102 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171bb2e007d1427ebf737bb9fa8b75a8" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:21.311205 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171bb2e007d1427ebf737bb9fa8b75a8" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:21.311424 11644 raft_consensus.cc:2454] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:21.311425 11249 raft_consensus.cc:2454] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:21.311898 11314 leader_election.cc:302] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:21.312135 12241 raft_consensus.cc:2781] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:21.312211 12241 raft_consensus.cc:482] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:21.312251 12241 raft_consensus.cc:3037] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.313086 12241 raft_consensus.cc:504] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.313398 12241 leader_election.cc:288] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:21.313676 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171bb2e007d1427ebf737bb9fa8b75a8" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:21.313697 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171bb2e007d1427ebf737bb9fa8b75a8" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:21.313786 11644 raft_consensus.cc:3037] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.313789 11249 raft_consensus.cc:3037] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.314574 11644 raft_consensus.cc:2454] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:21.314839 11314 leader_election.cc:302] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:21.315044 11249 raft_consensus.cc:2454] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:21.315162 12241 raft_consensus.cc:2781] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:21.315387 12241 raft_consensus.cc:686] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:21.315740 12241 consensus_queue.cc:237] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.318753 11104 catalog_manager.cc:5484] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:21.356828 11428 tablet.cc:2369] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:21.414638 12244 raft_consensus.cc:482] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.414789 12244 raft_consensus.cc:504] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.415913 12244 leader_election.cc:288] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:21.419624 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a53d7da1204a2a9cb0869330285284" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:21.419633 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a53d7da1204a2a9cb0869330285284" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:21.419833 11381 raft_consensus.cc:2454] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:21.419895 11512 raft_consensus.cc:2454] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:21.420236 11183 leader_election.cc:302] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:21.420473 12244 raft_consensus.cc:2781] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:21.420534 12244 raft_consensus.cc:482] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:21.420560 12244 raft_consensus.cc:3037] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.421365 12244 raft_consensus.cc:504] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.421739 12244 leader_election.cc:288] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:21.422066 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a53d7da1204a2a9cb0869330285284" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:21.422073 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a53d7da1204a2a9cb0869330285284" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:21.422169 11512 raft_consensus.cc:3037] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.422175 11381 raft_consensus.cc:3037] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.422921 11512 raft_consensus.cc:2454] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:21.422928 11381 raft_consensus.cc:2454] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:21.423192 11181 leader_election.cc:302] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:21.423358 12244 raft_consensus.cc:2781] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:21.423583 12244 raft_consensus.cc:686] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:21.423871 12244 consensus_queue.cc:237] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.426805 11104 catalog_manager.cc:5484] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:21.436354 11296 tablet.cc:2369] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:21.448379 12250 raft_consensus.cc:482] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.448518 12250 raft_consensus.cc:504] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.449550 12250 leader_election.cc:288] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:21.453265 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd620de3ef44eab9cd021874be0a3e4" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:21.453266 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd620de3ef44eab9cd021874be0a3e4" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:21.453397 11249 raft_consensus.cc:2454] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:21.453402 11512 raft_consensus.cc:2454] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:21.453778 11577 leader_election.cc:302] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
W20240501 19:56:21.453963 11691 tablet.cc:2369] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:21.454056 12250 raft_consensus.cc:2781] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:21.454120 12250 raft_consensus.cc:482] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:21.454161 12250 raft_consensus.cc:3037] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.455185 12250 raft_consensus.cc:504] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.455543 12250 leader_election.cc:288] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:21.455883 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd620de3ef44eab9cd021874be0a3e4" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:21.455893 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd620de3ef44eab9cd021874be0a3e4" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:21.455978 11249 raft_consensus.cc:3037] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.456001 11512 raft_consensus.cc:3037] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.456630 11249 raft_consensus.cc:2454] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:21.456713 11512 raft_consensus.cc:2454] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:21.456873 11577 leader_election.cc:302] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:21.457018 12250 raft_consensus.cc:2781] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:21.457208 12250 raft_consensus.cc:686] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:21.457496 12250 consensus_queue.cc:237] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.460376 11104 catalog_manager.cc:5484] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:21.584324 12264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:21.584595 12264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:21.587781 12264 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 19:56:21.700002 11249 raft_consensus.cc:1261] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:21.700785 12241 consensus_queue.cc:1035] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:21.701858 11248 raft_consensus.cc:1261] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:21.702131 11512 raft_consensus.cc:1261] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:21.702437 12262 consensus_queue.cc:1035] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:21.703104 12262 consensus_queue.cc:1035] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:21.704102 11512 raft_consensus.cc:1261] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:21.704578 12258 consensus_queue.cc:1035] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:21.705716 11381 raft_consensus.cc:1261] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:21.706993 12244 consensus_queue.cc:1035] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:21.713785 11644 raft_consensus.cc:1261] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:21.714260 12241 consensus_queue.cc:1035] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:21.721557 12300 mvcc.cc:205] Tried to move back new op lower bound from 7022974491436351488 to 7022974489871855616. Current Snapshot: MvccSnapshot[applied={T|T < 7022974491436351488}]
I20240501 19:56:21.724304 12296 mvcc.cc:205] Tried to move back new op lower bound from 7022974491444645888 to 7022974490452209664. Current Snapshot: MvccSnapshot[applied={T|T < 7022974491444645888}]
I20240501 19:56:21.729506 12301 mvcc.cc:205] Tried to move back new op lower bound from 7022974491444645888 to 7022974490452209664. Current Snapshot: MvccSnapshot[applied={T|T < 7022974491444645888}]
I20240501 19:56:21.736826 12294 mvcc.cc:205] Tried to move back new op lower bound from 7022974491445587968 to 7022974490314502144. Current Snapshot: MvccSnapshot[applied={T|T < 7022974491445587968}]
INSERT report
rows total: 2000
time total: 146.651 ms
time per row: 0.0733256 ms
I20240501 19:56:21.780148 11106 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:21.785929 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 169db26dfa3a4ec991c425c870e669eb (DEFAULT_TABLE table=rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.785908 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 169db26dfa3a4ec991c425c870e669eb (DEFAULT_TABLE table=rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.786237 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169db26dfa3a4ec991c425c870e669eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.786257 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169db26dfa3a4ec991c425c870e669eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.786356 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 169db26dfa3a4ec991c425c870e669eb (DEFAULT_TABLE table=rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.786593 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169db26dfa3a4ec991c425c870e669eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.786936 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 69facf86bba24f209c06afafc1ba8f7b (DEFAULT_TABLE table=rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.787168 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69facf86bba24f209c06afafc1ba8f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.787698 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 69facf86bba24f209c06afafc1ba8f7b (DEFAULT_TABLE table=rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.787909 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69facf86bba24f209c06afafc1ba8f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.788957 12322 tablet_bootstrap.cc:492] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:21.789896 12324 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:21.789919 12322 tablet_bootstrap.cc:654] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.790717 12323 tablet_bootstrap.cc:492] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:21.790840 12324 tablet_bootstrap.cc:654] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.791289 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 69facf86bba24f209c06afafc1ba8f7b (DEFAULT_TABLE table=rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.791561 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69facf86bba24f209c06afafc1ba8f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.791523 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 70fcac1c11404f51838dcb000f76d737 (DEFAULT_TABLE table=rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.791746 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fcac1c11404f51838dcb000f76d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.791743 12324 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:21.791874 12324 ts_tablet_manager.cc:1367] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:21.792245 12324 raft_consensus.cc:348] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.792344 12324 raft_consensus.cc:374] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.792372 12324 raft_consensus.cc:729] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.792487 12324 consensus_queue.cc:260] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.792711 12324 ts_tablet_manager.cc:1398] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:21.792970 12324 tablet_bootstrap.cc:492] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:21.793697 12324 tablet_bootstrap.cc:654] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.794533 12323 tablet_bootstrap.cc:654] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.794756 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 70fcac1c11404f51838dcb000f76d737 (DEFAULT_TABLE table=rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.794776 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 70fcac1c11404f51838dcb000f76d737 (DEFAULT_TABLE table=rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.794978 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fcac1c11404f51838dcb000f76d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.794981 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fcac1c11404f51838dcb000f76d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.795517 12322 tablet_bootstrap.cc:492] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:21.795603 12322 ts_tablet_manager.cc:1367] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:21.795650 12323 tablet_bootstrap.cc:492] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:21.795723 12323 ts_tablet_manager.cc:1367] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:21.796095 12322 raft_consensus.cc:348] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.796200 12322 raft_consensus.cc:374] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.796236 12322 raft_consensus.cc:729] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.796370 12322 consensus_queue.cc:260] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.796443 12323 raft_consensus.cc:348] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.796542 12323 raft_consensus.cc:374] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.796742 12323 raft_consensus.cc:729] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.796869 12323 consensus_queue.cc:260] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.797138 12323 ts_tablet_manager.cc:1398] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:21.797302 12323 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:21.797668 12322 ts_tablet_manager.cc:1398] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:21.797859 12322 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:21.798097 12323 tablet_bootstrap.cc:654] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.798844 12322 tablet_bootstrap.cc:654] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.800096 12324 tablet_bootstrap.cc:492] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:21.800191 12324 ts_tablet_manager.cc:1367] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240501 19:56:21.800151 12329 tablet_bootstrap.cc:492] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:21.800624 12324 raft_consensus.cc:348] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.800719 12324 raft_consensus.cc:374] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.800753 12324 raft_consensus.cc:729] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.800876 12324 consensus_queue.cc:260] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.800998 12329 tablet_bootstrap.cc:654] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.801115 12324 ts_tablet_manager.cc:1398] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:21.801255 12324 tablet_bootstrap.cc:492] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:21.801941 12323 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:21.802011 12329 tablet_bootstrap.cc:492] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:21.802018 12323 ts_tablet_manager.cc:1367] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:21.802073 12329 ts_tablet_manager.cc:1367] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:21.802412 12324 tablet_bootstrap.cc:654] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.802443 12323 raft_consensus.cc:348] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.802438 12329 raft_consensus.cc:348] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.802526 12323 raft_consensus.cc:374] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.802526 12329 raft_consensus.cc:374] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.802556 12329 raft_consensus.cc:729] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.802556 12323 raft_consensus.cc:729] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.802667 12329 consensus_queue.cc:260] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.802667 12323 consensus_queue.cc:260] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.802922 12323 ts_tablet_manager.cc:1398] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:21.802918 12329 ts_tablet_manager.cc:1398] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:21.804355 12322 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:21.804499 12322 ts_tablet_manager.cc:1367] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:21.804570 12324 tablet_bootstrap.cc:492] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:21.804668 12324 ts_tablet_manager.cc:1367] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:21.805030 12322 raft_consensus.cc:348] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.805109 12324 raft_consensus.cc:348] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.805225 12324 raft_consensus.cc:374] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.805181 12322 raft_consensus.cc:374] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.805275 12324 raft_consensus.cc:729] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.805351 12322 raft_consensus.cc:729] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.805531 12322 consensus_queue.cc:260] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.805574 12324 consensus_queue.cc:260] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.805944 12322 ts_tablet_manager.cc:1398] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:21.806169 12322 tablet_bootstrap.cc:492] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:21.806325 12324 ts_tablet_manager.cc:1398] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:21.806963 12322 tablet_bootstrap.cc:654] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.808115 12322 tablet_bootstrap.cc:492] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:21.808337 12322 ts_tablet_manager.cc:1367] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:21.808735 12322 raft_consensus.cc:348] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.808849 12322 raft_consensus.cc:374] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.808893 12322 raft_consensus.cc:729] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.809027 12322 consensus_queue.cc:260] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:21.809268 12322 ts_tablet_manager.cc:1398] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:21.823202 12241 raft_consensus.cc:482] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.823331 12241 raft_consensus.cc:504] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.823817 12241 leader_election.cc:288] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:21.824241 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:21.824399 11644 raft_consensus.cc:2454] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:21.824679 11314 leader_election.cc:302] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:21.824864 12241 raft_consensus.cc:2781] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:21.824923 12241 raft_consensus.cc:482] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:21.824956 12241 raft_consensus.cc:3037] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.825922 12241 raft_consensus.cc:504] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.826238 12241 leader_election.cc:288] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:21.826568 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:21.826668 11644 raft_consensus.cc:3037] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.827571 11644 raft_consensus.cc:2454] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:21.827630 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:21.827764 11512 raft_consensus.cc:2454] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:21.827838 11314 leader_election.cc:302] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:21.828127 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:21.828155 12241 raft_consensus.cc:2781] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:21.828230 12241 raft_consensus.cc:686] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:21.828234 11511 raft_consensus.cc:3037] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.828366 12241 consensus_queue.cc:237] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:21.829058 11511 raft_consensus.cc:2454] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:21.829900 11106 catalog_manager.cc:5484] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:21.878387 12310 raft_consensus.cc:482] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.878525 12310 raft_consensus.cc:504] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.878974 12310 leader_election.cc:288] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:21.879495 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69facf86bba24f209c06afafc1ba8f7b" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:21.879655 11511 raft_consensus.cc:2454] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:21.879951 11577 leader_election.cc:302] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:21.880105 12310 raft_consensus.cc:2781] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:21.880179 12310 raft_consensus.cc:482] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:21.880211 12310 raft_consensus.cc:3037] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.881178 12310 raft_consensus.cc:504] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.881548 12310 leader_election.cc:288] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:21.882025 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69facf86bba24f209c06afafc1ba8f7b" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:21.882148 11511 raft_consensus.cc:3037] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.882445 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69facf86bba24f209c06afafc1ba8f7b" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:21.882535 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69facf86bba24f209c06afafc1ba8f7b" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:21.882579 11381 raft_consensus.cc:2454] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:21.882649 11380 raft_consensus.cc:3037] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.883090 11511 raft_consensus.cc:2454] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:21.883358 11577 leader_election.cc:302] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:21.883530 12310 raft_consensus.cc:2781] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:21.883576 11380 raft_consensus.cc:2454] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:21.883617 12310 raft_consensus.cc:686] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:21.883750 12310 consensus_queue.cc:237] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:21.885449 11106 catalog_manager.cc:5484] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:22.042505 12241 raft_consensus.cc:482] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:22.042647 12241 raft_consensus.cc:504] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.043059 12241 leader_election.cc:288] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:22.043623 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fcac1c11404f51838dcb000f76d737" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:22.043690 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fcac1c11404f51838dcb000f76d737" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:22.043756 11511 raft_consensus.cc:2454] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:22.043860 11248 raft_consensus.cc:2454] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:22.044087 11313 leader_election.cc:302] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:22.044271 12241 raft_consensus.cc:2781] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:22.044325 12241 raft_consensus.cc:482] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:22.044350 12241 raft_consensus.cc:3037] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.045337 12241 raft_consensus.cc:504] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.045663 12241 leader_election.cc:288] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:22.046056 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fcac1c11404f51838dcb000f76d737" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:22.046038 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fcac1c11404f51838dcb000f76d737" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:22.046172 11248 raft_consensus.cc:3037] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.046172 11511 raft_consensus.cc:3037] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.047048 11511 raft_consensus.cc:2454] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:22.047048 11248 raft_consensus.cc:2454] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:22.047317 11313 leader_election.cc:302] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:22.047497 12241 raft_consensus.cc:2781] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:22.047581 12241 raft_consensus.cc:686] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:22.047711 12241 consensus_queue.cc:237] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.049422 11106 catalog_manager.cc:5484] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:22.194543 12339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:22.194888 12339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:22.199352 12339 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 19:56:22.290582 11644 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.290611 11511 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.291096 12310 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.291319 11380 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.292126 12254 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.294154 11512 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.294343 12310 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.294948 12254 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.295982 11248 raft_consensus.cc:1261] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.296396 12373 consensus_queue.cc:1035] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.296689 11510 raft_consensus.cc:1261] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.308935 12254 consensus_queue.cc:1035] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.955 ms
time per row: 0.0549776 ms
I20240501 19:56:22.353159 11106 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:22.358801 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 90d176eeeb5c4f9399f2a608cf98e319 (DEFAULT_TABLE table=rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.358984 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 90d176eeeb5c4f9399f2a608cf98e319 (DEFAULT_TABLE table=rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.359135 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d176eeeb5c4f9399f2a608cf98e319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.359234 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d176eeeb5c4f9399f2a608cf98e319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.359771 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 90d176eeeb5c4f9399f2a608cf98e319 (DEFAULT_TABLE table=rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.359874 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 7542b5a2a89848e5950ca8d4befc0497 (DEFAULT_TABLE table=rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.360045 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d176eeeb5c4f9399f2a608cf98e319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.360155 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7542b5a2a89848e5950ca8d4befc0497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.360271 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 7542b5a2a89848e5950ca8d4befc0497 (DEFAULT_TABLE table=rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.360491 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7542b5a2a89848e5950ca8d4befc0497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.362119 12397 tablet_bootstrap.cc:492] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:22.362653 11227 tablet_service.cc:1467] Processing CreateTablet for tablet b9b18195711b48f099a29c006cd89ce5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.362890 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b18195711b48f099a29c006cd89ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.363081 12397 tablet_bootstrap.cc:654] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.363036 11623 tablet_service.cc:1467] Processing CreateTablet for tablet b9b18195711b48f099a29c006cd89ce5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.363248 12398 tablet_bootstrap.cc:492] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:22.363274 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b18195711b48f099a29c006cd89ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.363533 11359 tablet_service.cc:1467] Processing CreateTablet for tablet b9b18195711b48f099a29c006cd89ce5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.363767 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b18195711b48f099a29c006cd89ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.365034 12398 tablet_bootstrap.cc:654] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.365419 12399 tablet_bootstrap.cc:492] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:22.366182 12399 tablet_bootstrap.cc:654] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.366371 12398 tablet_bootstrap.cc:492] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:22.366461 12398 ts_tablet_manager.cc:1367] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:22.366880 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 7542b5a2a89848e5950ca8d4befc0497 (DEFAULT_TABLE table=rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.366925 12398 raft_consensus.cc:348] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.367028 12398 raft_consensus.cc:374] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.367058 12398 raft_consensus.cc:729] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.367123 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7542b5a2a89848e5950ca8d4befc0497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.367208 12398 consensus_queue.cc:260] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.367424 12399 tablet_bootstrap.cc:492] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:22.367580 12399 ts_tablet_manager.cc:1367] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:22.367959 12399 raft_consensus.cc:348] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.368448 12398 ts_tablet_manager.cc:1398] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:22.368045 12399 raft_consensus.cc:374] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.368834 12399 raft_consensus.cc:729] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.368978 12399 consensus_queue.cc:260] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.369311 12399 ts_tablet_manager.cc:1398] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:22.369464 12399 tablet_bootstrap.cc:492] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:22.370867 12399 tablet_bootstrap.cc:654] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.371661 12396 tablet_bootstrap.cc:492] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:22.371913 12399 tablet_bootstrap.cc:492] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:22.371985 12399 ts_tablet_manager.cc:1367] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:22.372320 12399 raft_consensus.cc:348] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.372411 12399 raft_consensus.cc:374] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.372437 12399 raft_consensus.cc:729] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.372483 12396 tablet_bootstrap.cc:654] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.372532 12399 consensus_queue.cc:260] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.372746 12399 ts_tablet_manager.cc:1398] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:22.372887 12399 tablet_bootstrap.cc:492] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:22.373533 12399 tablet_bootstrap.cc:654] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.373551 12397 tablet_bootstrap.cc:492] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:22.373636 12397 ts_tablet_manager.cc:1367] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:22.374068 12397 raft_consensus.cc:348] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.374181 12397 raft_consensus.cc:374] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.374220 12397 raft_consensus.cc:729] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.374342 12397 consensus_queue.cc:260] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.374612 12397 ts_tablet_manager.cc:1398] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:22.374881 12399 tablet_bootstrap.cc:492] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:22.374929 12397 tablet_bootstrap.cc:492] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:22.374951 12399 ts_tablet_manager.cc:1367] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:22.375365 12399 raft_consensus.cc:348] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.375483 12399 raft_consensus.cc:374] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.375516 12399 raft_consensus.cc:729] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.375743 12397 tablet_bootstrap.cc:654] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.375736 12399 consensus_queue.cc:260] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.376789 12399 ts_tablet_manager.cc:1398] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:22.377519 12396 tablet_bootstrap.cc:492] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:22.377612 12396 ts_tablet_manager.cc:1367] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:22.378067 12396 raft_consensus.cc:348] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.378183 12396 raft_consensus.cc:374] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.378216 12396 raft_consensus.cc:729] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.378360 12396 consensus_queue.cc:260] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.378641 12396 ts_tablet_manager.cc:1398] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.379230 12397 tablet_bootstrap.cc:492] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:22.379288 12396 tablet_bootstrap.cc:492] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:22.379308 12397 ts_tablet_manager.cc:1367] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:22.379770 12397 raft_consensus.cc:348] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.379871 12397 raft_consensus.cc:374] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.379909 12397 raft_consensus.cc:729] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.380033 12397 consensus_queue.cc:260] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.380357 12397 ts_tablet_manager.cc:1398] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.380517 12397 tablet_bootstrap.cc:492] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:22.380870 12396 tablet_bootstrap.cc:654] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.381745 12397 tablet_bootstrap.cc:654] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.382839 12397 tablet_bootstrap.cc:492] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:22.382930 12397 ts_tablet_manager.cc:1367] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:22.382973 12396 tablet_bootstrap.cc:492] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:22.383051 12396 ts_tablet_manager.cc:1367] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:22.383276 12397 raft_consensus.cc:348] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.383360 12397 raft_consensus.cc:374] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.383384 12397 raft_consensus.cc:729] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.383508 12397 consensus_queue.cc:260] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.383730 12397 ts_tablet_manager.cc:1398] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.383668 12396 raft_consensus.cc:348] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.383836 12396 raft_consensus.cc:374] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.383915 12396 raft_consensus.cc:729] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.384047 12396 consensus_queue.cc:260] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.384305 12396 ts_tablet_manager.cc:1398] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.416301 12262 raft_consensus.cc:482] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:22.416433 12262 raft_consensus.cc:504] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.416867 12262 leader_election.cc:288] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:22.417238 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d176eeeb5c4f9399f2a608cf98e319" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:22.417310 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d176eeeb5c4f9399f2a608cf98e319" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:22.417373 11380 raft_consensus.cc:2454] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:22.417474 11248 raft_consensus.cc:2454] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:22.417661 11579 leader_election.cc:302] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:22.417848 12262 raft_consensus.cc:2781] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:22.417896 12262 raft_consensus.cc:482] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:22.417922 12262 raft_consensus.cc:3037] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.418722 12262 raft_consensus.cc:504] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.419032 12262 leader_election.cc:288] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:22.419420 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d176eeeb5c4f9399f2a608cf98e319" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:22.419420 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d176eeeb5c4f9399f2a608cf98e319" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:22.419569 11248 raft_consensus.cc:3037] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.419569 11380 raft_consensus.cc:3037] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.420504 11380 raft_consensus.cc:2454] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:22.420504 11248 raft_consensus.cc:2454] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:22.420810 11577 leader_election.cc:302] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:22.421070 12262 raft_consensus.cc:2781] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:22.421160 12262 raft_consensus.cc:686] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:22.421295 12262 consensus_queue.cc:237] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.422941 11100 catalog_manager.cc:5484] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:22.478677 12262 raft_consensus.cc:482] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:22.478822 12262 raft_consensus.cc:504] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.479239 12262 leader_election.cc:288] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:22.479768 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b18195711b48f099a29c006cd89ce5" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:22.479789 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b18195711b48f099a29c006cd89ce5" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:22.479950 11248 raft_consensus.cc:2454] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:22.479950 11380 raft_consensus.cc:2454] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:22.480262 11577 leader_election.cc:302] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:22.480450 12262 raft_consensus.cc:2781] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:22.480515 12262 raft_consensus.cc:482] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:22.480540 12262 raft_consensus.cc:3037] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.481303 12262 raft_consensus.cc:504] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.481652 12262 leader_election.cc:288] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:22.482000 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b18195711b48f099a29c006cd89ce5" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:22.482002 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b18195711b48f099a29c006cd89ce5" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:22.482124 11248 raft_consensus.cc:3037] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.482124 11380 raft_consensus.cc:3037] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.483022 11380 raft_consensus.cc:2454] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:22.483023 11248 raft_consensus.cc:2454] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:22.483319 11579 leader_election.cc:302] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:22.483527 12262 raft_consensus.cc:2781] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:22.483623 12262 raft_consensus.cc:686] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:22.483759 12262 consensus_queue.cc:237] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.485435 11106 catalog_manager.cc:5484] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:22.541972 12244 raft_consensus.cc:482] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:22.542117 12244 raft_consensus.cc:504] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.542629 12244 leader_election.cc:288] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:22.543105 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7542b5a2a89848e5950ca8d4befc0497" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:22.543262 11511 raft_consensus.cc:2454] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:22.543623 11181 leader_election.cc:302] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:22.543817 12244 raft_consensus.cc:2781] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:22.543924 12244 raft_consensus.cc:482] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:22.543972 12244 raft_consensus.cc:3037] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.544883 12244 raft_consensus.cc:504] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.545296 12244 leader_election.cc:288] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:22.545614 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7542b5a2a89848e5950ca8d4befc0497" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:22.545732 11511 raft_consensus.cc:3037] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.546694 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7542b5a2a89848e5950ca8d4befc0497" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:22.546775 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7542b5a2a89848e5950ca8d4befc0497" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:22.546841 11644 raft_consensus.cc:2454] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:22.546881 11511 raft_consensus.cc:2454] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:22.547181 11181 leader_election.cc:302] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:22.547425 12244 raft_consensus.cc:2781] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:22.547613 12244 raft_consensus.cc:686] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:22.547755 12244 consensus_queue.cc:237] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:22.549477 11106 catalog_manager.cc:5484] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:22.703291 12414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:22.703599 12414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:22.706809 12414 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 19:56:22.799248 11248 raft_consensus.cc:1261] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.800105 11380 raft_consensus.cc:1261] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.800635 12310 consensus_queue.cc:1035] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.800935 12310 consensus_queue.cc:1035] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.801908 11381 raft_consensus.cc:1261] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.802970 11511 raft_consensus.cc:1261] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.803622 11643 raft_consensus.cc:3037] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.804723 11643 raft_consensus.cc:1261] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.805096 11247 raft_consensus.cc:1261] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.805569 12244 consensus_queue.cc:1035] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.808324 12443 consensus_queue.cc:1035] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.815057 12244 consensus_queue.cc:1035] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.817315 12262 consensus_queue.cc:1035] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.244 ms
time per row: 0.0596221 ms
I20240501 19:56:22.867341 11106 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:22.872445 11492 tablet_service.cc:1467] Processing CreateTablet for tablet c92c6475f5ed4529b1efd12b6d495a11 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.872555 11359 tablet_service.cc:1467] Processing CreateTablet for tablet c92c6475f5ed4529b1efd12b6d495a11 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.872790 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92c6475f5ed4529b1efd12b6d495a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.872851 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92c6475f5ed4529b1efd12b6d495a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.872881 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 04932055883c412f944e839982815ab6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.873114 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04932055883c412f944e839982815ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.874668 12455 tablet_bootstrap.cc:492] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:22.874863 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 00b7dd445e2846d5a0e1277e160d46bc (DEFAULT_TABLE table=rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.874998 12397 tablet_bootstrap.cc:492] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:22.875084 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b7dd445e2846d5a0e1277e160d46bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.875644 12455 tablet_bootstrap.cc:654] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.875891 12397 tablet_bootstrap.cc:654] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.876658 12455 tablet_bootstrap.cc:492] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:22.876746 12455 ts_tablet_manager.cc:1367] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:22.876960 12396 tablet_bootstrap.cc:492] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:22.877107 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 04932055883c412f944e839982815ab6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.877362 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04932055883c412f944e839982815ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.877450 12397 tablet_bootstrap.cc:492] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:22.877516 12397 ts_tablet_manager.cc:1367] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:22.877523 12455 raft_consensus.cc:348] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.877621 12455 raft_consensus.cc:374] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.877651 12455 raft_consensus.cc:729] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.877753 12396 tablet_bootstrap.cc:654] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.877769 12455 consensus_queue.cc:260] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.877888 12397 raft_consensus.cc:348] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.877975 12397 raft_consensus.cc:374] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.878000 12397 raft_consensus.cc:729] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.878113 12455 ts_tablet_manager.cc:1398] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:22.878108 12397 consensus_queue.cc:260] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.878345 12397 ts_tablet_manager.cc:1398] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.878515 12397 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:22.878880 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 00b7dd445e2846d5a0e1277e160d46bc (DEFAULT_TABLE table=rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.879067 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b7dd445e2846d5a0e1277e160d46bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.879273 12397 tablet_bootstrap.cc:654] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.879431 12459 tablet_bootstrap.cc:492] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:22.877107 11229 tablet_service.cc:1467] Processing CreateTablet for tablet c92c6475f5ed4529b1efd12b6d495a11 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.879868 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92c6475f5ed4529b1efd12b6d495a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.880245 12459 tablet_bootstrap.cc:654] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.880914 12455 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:22.881539 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 00b7dd445e2846d5a0e1277e160d46bc (DEFAULT_TABLE table=rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.881769 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b7dd445e2846d5a0e1277e160d46bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.881773 12455 tablet_bootstrap.cc:654] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.882423 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 04932055883c412f944e839982815ab6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.882620 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04932055883c412f944e839982815ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.882674 12459 tablet_bootstrap.cc:492] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:22.882743 12459 ts_tablet_manager.cc:1367] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:22.884532 12459 raft_consensus.cc:348] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.884650 12459 raft_consensus.cc:374] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.884681 12459 raft_consensus.cc:729] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.884799 12397 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:22.884878 12397 ts_tablet_manager.cc:1367] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:22.884832 12459 consensus_queue.cc:260] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.885191 12459 ts_tablet_manager.cc:1398] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:22.885280 12397 raft_consensus.cc:348] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:22.885360 12459 tablet_bootstrap.cc:492] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:22.885386 12397 raft_consensus.cc:374] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.885419 12397 raft_consensus.cc:729] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.885547 12397 consensus_queue.cc:260] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:22.885900 12396 tablet_bootstrap.cc:492] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:22.886116 12396 ts_tablet_manager.cc:1367] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:22.886152 12459 tablet_bootstrap.cc:654] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.886574 12396 raft_consensus.cc:348] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.886696 12396 raft_consensus.cc:374] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.886726 12396 raft_consensus.cc:729] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.886857 12396 consensus_queue.cc:260] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.887115 12396 ts_tablet_manager.cc:1398] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.887202 12397 ts_tablet_manager.cc:1398] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:22.887284 12396 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:22.887622 12459 tablet_bootstrap.cc:492] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:22.887746 12459 ts_tablet_manager.cc:1367] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:22.888191 12459 raft_consensus.cc:348] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.888358 12459 raft_consensus.cc:374] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.888432 12459 raft_consensus.cc:729] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.888604 12459 consensus_queue.cc:260] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.888782 12396 tablet_bootstrap.cc:654] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.888983 12459 ts_tablet_manager.cc:1398] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.890149 12396 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:22.890231 12396 ts_tablet_manager.cc:1367] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:22.890621 12396 raft_consensus.cc:348] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:22.890717 12396 raft_consensus.cc:374] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.890748 12396 raft_consensus.cc:729] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.890868 12396 consensus_queue.cc:260] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:22.891122 12396 ts_tablet_manager.cc:1398] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.892608 12455 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:22.892709 12455 ts_tablet_manager.cc:1367] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240501 19:56:22.893154 12455 raft_consensus.cc:348] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:22.893247 12455 raft_consensus.cc:374] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.893267 12455 raft_consensus.cc:729] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.893380 12455 consensus_queue.cc:260] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:22.893580 12455 ts_tablet_manager.cc:1398] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:22.893710 12455 tablet_bootstrap.cc:492] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:22.894498 12455 tablet_bootstrap.cc:654] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.895725 12455 tablet_bootstrap.cc:492] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:22.895825 12455 ts_tablet_manager.cc:1367] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:22.896231 12455 raft_consensus.cc:348] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.896342 12455 raft_consensus.cc:374] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.896384 12455 raft_consensus.cc:729] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.896553 12455 consensus_queue.cc:260] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:22.896816 12455 ts_tablet_manager.cc:1398] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.935914 12243 raft_consensus.cc:482] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:22.936043 12243 raft_consensus.cc:504] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:22.937001 12243 leader_election.cc:288] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:22.940546 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b7dd445e2846d5a0e1277e160d46bc" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:22.940696 11643 raft_consensus.cc:2454] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:22.940724 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b7dd445e2846d5a0e1277e160d46bc" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:22.940871 11380 raft_consensus.cc:2454] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:22.941080 11446 leader_election.cc:302] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:22.941351 12243 raft_consensus.cc:2781] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:22.941408 12243 raft_consensus.cc:482] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:22.941434 12243 raft_consensus.cc:3037] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.942307 12243 raft_consensus.cc:504] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:22.942641 12243 leader_election.cc:288] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:22.942967 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b7dd445e2846d5a0e1277e160d46bc" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:22.942972 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b7dd445e2846d5a0e1277e160d46bc" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:22.943069 11380 raft_consensus.cc:3037] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.943076 11643 raft_consensus.cc:3037] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.943827 11380 raft_consensus.cc:2454] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:22.943914 11643 raft_consensus.cc:2454] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:22.944103 11447 leader_election.cc:302] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:22.944254 12243 raft_consensus.cc:2781] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:22.944480 12243 raft_consensus.cc:686] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:22.944801 12243 consensus_queue.cc:237] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:22.947180 11101 catalog_manager.cc:5484] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:23.142377 12451 raft_consensus.cc:482] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:23.142513 12451 raft_consensus.cc:504] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.142982 12451 leader_election.cc:288] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:23.143469 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92c6475f5ed4529b1efd12b6d495a11" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:23.143481 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92c6475f5ed4529b1efd12b6d495a11" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:23.143604 11511 raft_consensus.cc:2454] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:23.143630 11380 raft_consensus.cc:2454] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:23.143915 11181 leader_election.cc:302] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:23.144084 12451 raft_consensus.cc:2781] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:23.144151 12451 raft_consensus.cc:482] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:23.144177 12451 raft_consensus.cc:3037] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.144999 12451 raft_consensus.cc:504] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.145303 12451 leader_election.cc:288] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:23.145607 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92c6475f5ed4529b1efd12b6d495a11" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:23.145640 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92c6475f5ed4529b1efd12b6d495a11" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:23.145735 11380 raft_consensus.cc:3037] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.145735 11511 raft_consensus.cc:3037] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.146611 11511 raft_consensus.cc:2454] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:23.146611 11380 raft_consensus.cc:2454] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:23.146903 11183 leader_election.cc:302] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:23.147106 12451 raft_consensus.cc:2781] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:23.147184 12451 raft_consensus.cc:686] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:23.147311 12451 consensus_queue.cc:237] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.149006 11101 catalog_manager.cc:5484] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:23.250691 12243 raft_consensus.cc:482] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:23.250847 12243 raft_consensus.cc:504] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.251336 12243 leader_election.cc:288] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:23.251717 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04932055883c412f944e839982815ab6" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:23.251865 11643 raft_consensus.cc:2454] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:23.252120 11446 leader_election.cc:302] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:23.252301 12243 raft_consensus.cc:2781] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:23.252358 12243 raft_consensus.cc:482] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:23.252384 12243 raft_consensus.cc:3037] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.253186 12243 raft_consensus.cc:504] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.253530 12243 leader_election.cc:288] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:23.253870 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04932055883c412f944e839982815ab6" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:23.253986 11643 raft_consensus.cc:3037] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.254861 11643 raft_consensus.cc:2454] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:23.255165 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04932055883c412f944e839982815ab6" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:23.255139 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04932055883c412f944e839982815ab6" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:23.255308 11248 raft_consensus.cc:3037] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.256065 11248 raft_consensus.cc:2454] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:23.256274 11446 leader_election.cc:302] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:23.256476 12243 raft_consensus.cc:2781] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:23.256564 12243 raft_consensus.cc:686] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:23.256810 12243 consensus_queue.cc:237] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.258932 11101 catalog_manager.cc:5484] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:23.326164 12243 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:23.329607 12474 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:23.442392 12476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:23.442652 12476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:23.445835 12476 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 19:56:23.537643 11511 raft_consensus.cc:1261] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:23.538022 11380 raft_consensus.cc:1261] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:23.538440 12412 consensus_queue.cc:1035] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:23.543540 11248 raft_consensus.cc:1261] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:23.543743 11643 raft_consensus.cc:1261] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:23.544291 12474 consensus_queue.cc:1035] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:23.544586 12469 consensus_queue.cc:1035] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:23.545431 12451 consensus_queue.cc:1035] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.584 ms
time per row: 0.0592919 ms
I20240501 19:56:23.604274 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:23.609959 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 9e327331ed1a4790af392bf83e6a7cd2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:23.610065 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 9e327331ed1a4790af392bf83e6a7cd2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:23.610320 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e327331ed1a4790af392bf83e6a7cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.610378 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e327331ed1a4790af392bf83e6a7cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.610846 11623 tablet_service.cc:1467] Processing CreateTablet for tablet fc32d7a811f741f4927b60b0bb9b6046 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:23.611130 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc32d7a811f741f4927b60b0bb9b6046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.610848 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc636fe78c14563aac456322e1eda5c (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:23.611903 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc636fe78c14563aac456322e1eda5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.613021 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 9e327331ed1a4790af392bf83e6a7cd2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:23.613272 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e327331ed1a4790af392bf83e6a7cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.613576 12532 tablet_bootstrap.cc:492] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:23.614328 12530 tablet_bootstrap.cc:492] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:23.614696 12532 tablet_bootstrap.cc:654] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:23.614718 11359 tablet_service.cc:1467] Processing CreateTablet for tablet fc32d7a811f741f4927b60b0bb9b6046 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:23.614970 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc32d7a811f741f4927b60b0bb9b6046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.615199 12530 tablet_bootstrap.cc:654] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:23.615744 12532 tablet_bootstrap.cc:492] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:23.615850 12532 ts_tablet_manager.cc:1367] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:23.616236 12530 tablet_bootstrap.cc:492] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:23.616334 12530 ts_tablet_manager.cc:1367] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:23.616297 12532 raft_consensus.cc:348] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.616394 12532 raft_consensus.cc:374] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:23.616432 12532 raft_consensus.cc:729] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.616565 12532 consensus_queue.cc:260] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.616771 12530 raft_consensus.cc:348] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:23.616840 12532 ts_tablet_manager.cc:1398] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:23.616878 12530 raft_consensus.cc:374] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:23.616847 11229 tablet_service.cc:1467] Processing CreateTablet for tablet fc32d7a811f741f4927b60b0bb9b6046 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:23.616914 12530 raft_consensus.cc:729] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.617017 12532 tablet_bootstrap.cc:492] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:23.617072 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc32d7a811f741f4927b60b0bb9b6046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.617034 12530 consensus_queue.cc:260] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:23.617337 12530 ts_tablet_manager.cc:1398] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:23.617841 12532 tablet_bootstrap.cc:654] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:23.618607 12534 tablet_bootstrap.cc:492] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:23.618897 12530 tablet_bootstrap.cc:492] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:23.619094 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc636fe78c14563aac456322e1eda5c (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:23.619314 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc636fe78c14563aac456322e1eda5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.619812 12534 tablet_bootstrap.cc:654] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:23.619851 12530 tablet_bootstrap.cc:654] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:23.620790 12532 tablet_bootstrap.cc:492] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:23.620873 12532 ts_tablet_manager.cc:1367] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:23.621006 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc636fe78c14563aac456322e1eda5c (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:23.621217 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc636fe78c14563aac456322e1eda5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:23.621318 12532 raft_consensus.cc:348] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.621420 12532 raft_consensus.cc:374] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:23.621452 12532 raft_consensus.cc:729] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.621598 12532 consensus_queue.cc:260] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.621860 12532 ts_tablet_manager.cc:1398] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:23.622110 12531 tablet_bootstrap.cc:492] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:23.622907 12530 tablet_bootstrap.cc:492] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:23.622998 12530 ts_tablet_manager.cc:1367] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:23.623396 12531 tablet_bootstrap.cc:654] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:23.623415 12530 raft_consensus.cc:348] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.623546 12530 raft_consensus.cc:374] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:23.623587 12530 raft_consensus.cc:729] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.623723 12530 consensus_queue.cc:260] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.624024 12530 ts_tablet_manager.cc:1398] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:23.624182 12530 tablet_bootstrap.cc:492] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:23.624588 12531 tablet_bootstrap.cc:492] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:23.624589 12534 tablet_bootstrap.cc:492] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:23.624675 12534 ts_tablet_manager.cc:1367] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:23.624677 12531 ts_tablet_manager.cc:1367] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:23.625043 12530 tablet_bootstrap.cc:654] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:23.625098 12534 raft_consensus.cc:348] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:23.625119 12531 raft_consensus.cc:348] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:23.625190 12534 raft_consensus.cc:374] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:23.625221 12534 raft_consensus.cc:729] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.625223 12531 raft_consensus.cc:374] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:23.625257 12531 raft_consensus.cc:729] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.625372 12534 consensus_queue.cc:260] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:23.625389 12531 consensus_queue.cc:260] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:23.625645 12531 ts_tablet_manager.cc:1398] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:23.625813 12531 tablet_bootstrap.cc:492] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:23.626566 12534 ts_tablet_manager.cc:1398] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:23.626627 12531 tablet_bootstrap.cc:654] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:23.627105 12530 tablet_bootstrap.cc:492] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:23.627113 12534 tablet_bootstrap.cc:492] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:23.628028 12530 ts_tablet_manager.cc:1367] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:56:23.628379 12534 tablet_bootstrap.cc:654] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:23.628566 12530 raft_consensus.cc:348] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.628676 12530 raft_consensus.cc:374] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:23.628711 12530 raft_consensus.cc:729] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.628841 12530 consensus_queue.cc:260] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.629287 12530 ts_tablet_manager.cc:1398] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:23.629526 12534 tablet_bootstrap.cc:492] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:23.629616 12534 ts_tablet_manager.cc:1367] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:23.630091 12534 raft_consensus.cc:348] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.630206 12534 raft_consensus.cc:374] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:23.630247 12534 raft_consensus.cc:729] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.630374 12534 consensus_queue.cc:260] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.630653 12534 ts_tablet_manager.cc:1398] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:23.630901 12531 tablet_bootstrap.cc:492] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:23.630997 12531 ts_tablet_manager.cc:1367] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 19:56:23.631430 12531 raft_consensus.cc:348] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.631567 12531 raft_consensus.cc:374] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:23.631588 12531 raft_consensus.cc:729] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.631703 12531 consensus_queue.cc:260] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.631978 12531 ts_tablet_manager.cc:1398] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:23.685446 12451 raft_consensus.cc:482] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:23.685573 12451 raft_consensus.cc:504] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.686005 12451 leader_election.cc:288] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:23.686421 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc32d7a811f741f4927b60b0bb9b6046" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:23.686429 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc32d7a811f741f4927b60b0bb9b6046" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:23.686558 11643 raft_consensus.cc:2454] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:23.686563 11380 raft_consensus.cc:2454] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:23.687240 11182 leader_election.cc:302] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:23.687476 12451 raft_consensus.cc:2781] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:23.687546 12451 raft_consensus.cc:482] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:23.687583 12451 raft_consensus.cc:3037] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.688375 12451 raft_consensus.cc:504] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.688722 12451 leader_election.cc:288] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:23.688858 12513 raft_consensus.cc:482] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:23.688958 12513 raft_consensus.cc:504] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:23.689090 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc32d7a811f741f4927b60b0bb9b6046" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:23.689149 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc32d7a811f741f4927b60b0bb9b6046" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:23.689213 11643 raft_consensus.cc:3037] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.689275 11380 raft_consensus.cc:3037] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.689473 12513 leader_election.cc:288] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:23.689767 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e327331ed1a4790af392bf83e6a7cd2" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:23.689875 11381 raft_consensus.cc:2454] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:23.689927 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e327331ed1a4790af392bf83e6a7cd2" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:23.690039 11248 raft_consensus.cc:2454] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:23.690176 11447 leader_election.cc:302] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:23.690212 11643 raft_consensus.cc:2454] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:23.690282 11380 raft_consensus.cc:2454] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:23.690351 12513 raft_consensus.cc:2781] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:23.690397 12513 raft_consensus.cc:482] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:23.690435 12513 raft_consensus.cc:3037] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.690491 11182 leader_election.cc:302] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:23.690635 12451 raft_consensus.cc:2781] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:23.690707 12451 raft_consensus.cc:686] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:23.690847 12451 consensus_queue.cc:237] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.691416 12513 raft_consensus.cc:504] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:23.691794 12513 leader_election.cc:288] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:23.692137 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e327331ed1a4790af392bf83e6a7cd2" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:23.692238 11248 raft_consensus.cc:3037] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.692242 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e327331ed1a4790af392bf83e6a7cd2" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:23.692334 11380 raft_consensus.cc:3037] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.692922 11106 catalog_manager.cc:5484] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:23.693089 11248 raft_consensus.cc:2454] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:23.693089 11380 raft_consensus.cc:2454] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:23.693358 11445 leader_election.cc:302] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:23.693522 12513 raft_consensus.cc:2781] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:23.693620 12513 raft_consensus.cc:686] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:23.693742 12513 consensus_queue.cc:237] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:23.695362 11099 catalog_manager.cc:5484] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:23.788460 12443 raft_consensus.cc:482] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:23.788609 12443 raft_consensus.cc:504] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.789069 12443 leader_election.cc:288] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:23.789556 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc636fe78c14563aac456322e1eda5c" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:23.789584 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc636fe78c14563aac456322e1eda5c" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:23.789696 11511 raft_consensus.cc:2454] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:23.789742 11248 raft_consensus.cc:2454] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:23.790055 11577 leader_election.cc:302] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:23.790256 12443 raft_consensus.cc:2781] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:23.790318 12443 raft_consensus.cc:482] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:23.790360 12443 raft_consensus.cc:3037] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.791185 12443 raft_consensus.cc:504] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.791591 12443 leader_election.cc:288] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:23.791961 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc636fe78c14563aac456322e1eda5c" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:23.792030 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc636fe78c14563aac456322e1eda5c" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:23.792085 11511 raft_consensus.cc:3037] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.792152 11248 raft_consensus.cc:3037] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:23.792829 11511 raft_consensus.cc:2454] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:23.792908 11248 raft_consensus.cc:2454] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:23.793144 11577 leader_election.cc:302] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:23.793329 12443 raft_consensus.cc:2781] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:23.793550 12443 raft_consensus.cc:686] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:23.793695 12443 consensus_queue.cc:237] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:23.795513 11099 catalog_manager.cc:5484] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:23.917711 12547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:23.917982 12547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:23.921092 12547 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 19:56:24.013732 11248 raft_consensus.cc:1261] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.013860 11249 raft_consensus.cc:1261] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.013859 11511 raft_consensus.cc:1261] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.014357 11380 raft_consensus.cc:1261] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.014508 12545 consensus_queue.cc:1035] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.014958 12443 consensus_queue.cc:1035] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.015105 12513 consensus_queue.cc:1035] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.016062 12513 consensus_queue.cc:1035] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.024019 11380 raft_consensus.cc:1261] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.027714 12412 consensus_queue.cc:1035] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.028633 11644 raft_consensus.cc:1261] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.030570 12412 consensus_queue.cc:1035] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.062813 12506 mvcc.cc:205] Tried to move back new op lower bound from 7022974500956200960 to 7022974499598991360. Current Snapshot: MvccSnapshot[applied={T|T < 7022974500937428992}]
INSERT report
rows total: 2000
time total: 129.813 ms
time per row: 0.0649063 ms
I20240501 19:56:24.093135 11098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:24.099067 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 01a10926bce14cf1bd12f21287e08c89 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.099274 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 858f60ea9d814c69ac626cd22c9ea994 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.099397 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a10926bce14cf1bd12f21287e08c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.099606 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858f60ea9d814c69ac626cd22c9ea994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.099629 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 01a10926bce14cf1bd12f21287e08c89 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.099973 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 2c86f45702e94c0e898e4e81148f2098 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.100212 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c86f45702e94c0e898e4e81148f2098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.100437 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 858f60ea9d814c69ac626cd22c9ea994 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.100649 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858f60ea9d814c69ac626cd22c9ea994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.101642 12531 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:24.102427 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 2c86f45702e94c0e898e4e81148f2098 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.102533 12531 tablet_bootstrap.cc:654] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.102608 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c86f45702e94c0e898e4e81148f2098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.102663 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 01a10926bce14cf1bd12f21287e08c89 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.102918 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a10926bce14cf1bd12f21287e08c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.103157 12530 tablet_bootstrap.cc:492] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:24.103969 12530 tablet_bootstrap.cc:654] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.104158 12534 tablet_bootstrap.cc:492] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:24.104904 12534 tablet_bootstrap.cc:654] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.104943 12531 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:24.104960 12532 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:24.105026 12531 ts_tablet_manager.cc:1367] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:24.105504 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a10926bce14cf1bd12f21287e08c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.105458 12531 raft_consensus.cc:348] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.105572 12531 raft_consensus.cc:374] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.105602 12531 raft_consensus.cc:729] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.105734 12532 tablet_bootstrap.cc:654] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.105718 12531 consensus_queue.cc:260] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.105758 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 858f60ea9d814c69ac626cd22c9ea994 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.105978 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858f60ea9d814c69ac626cd22c9ea994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.105985 12531 ts_tablet_manager.cc:1398] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.106142 12531 tablet_bootstrap.cc:492] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:24.106954 12531 tablet_bootstrap.cc:654] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.107864 12530 tablet_bootstrap.cc:492] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:24.107975 12530 ts_tablet_manager.cc:1367] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:24.108377 12530 raft_consensus.cc:348] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.108505 12530 raft_consensus.cc:374] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.108546 12530 raft_consensus.cc:729] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.108671 12530 consensus_queue.cc:260] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.108958 12530 ts_tablet_manager.cc:1398] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:24.109627 12531 tablet_bootstrap.cc:492] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:24.109722 12531 ts_tablet_manager.cc:1367] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:24.110183 12531 raft_consensus.cc:348] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.110280 12531 raft_consensus.cc:374] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.110307 12531 raft_consensus.cc:729] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.110415 12531 consensus_queue.cc:260] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.110512 12532 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:24.110534 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 2c86f45702e94c0e898e4e81148f2098 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.110594 12532 ts_tablet_manager.cc:1367] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:24.110661 12531 ts_tablet_manager.cc:1398] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:24.110766 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c86f45702e94c0e898e4e81148f2098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.110798 12531 tablet_bootstrap.cc:492] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:24.111014 12532 raft_consensus.cc:348] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.111119 12532 raft_consensus.cc:374] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.111151 12532 raft_consensus.cc:729] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.111258 12532 consensus_queue.cc:260] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.111557 12532 ts_tablet_manager.cc:1398] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.111642 12531 tablet_bootstrap.cc:654] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.111712 12532 tablet_bootstrap.cc:492] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:24.112533 12532 tablet_bootstrap.cc:654] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.113622 12534 tablet_bootstrap.cc:492] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:24.113706 12534 ts_tablet_manager.cc:1367] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 19:56:24.114122 12534 raft_consensus.cc:348] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.114218 12534 raft_consensus.cc:374] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.114245 12534 raft_consensus.cc:729] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.114356 12534 consensus_queue.cc:260] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.114626 12534 ts_tablet_manager.cc:1398] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:24.114813 12534 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:24.114985 12531 tablet_bootstrap.cc:492] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:24.115105 12531 ts_tablet_manager.cc:1367] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:24.115610 12534 tablet_bootstrap.cc:654] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.115600 12531 raft_consensus.cc:348] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.115680 12531 raft_consensus.cc:374] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.115720 12531 raft_consensus.cc:729] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.115841 12531 consensus_queue.cc:260] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.116107 12531 ts_tablet_manager.cc:1398] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.116619 12534 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:24.116703 12534 ts_tablet_manager.cc:1367] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:24.117045 12534 raft_consensus.cc:348] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.117144 12534 raft_consensus.cc:374] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.117179 12534 raft_consensus.cc:729] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.117287 12534 consensus_queue.cc:260] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.117482 12532 tablet_bootstrap.cc:492] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:24.117568 12532 ts_tablet_manager.cc:1367] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:24.117722 12534 ts_tablet_manager.cc:1398] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.117889 12534 tablet_bootstrap.cc:492] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:24.117998 12532 raft_consensus.cc:348] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.118105 12532 raft_consensus.cc:374] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.118139 12532 raft_consensus.cc:729] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.118266 12532 consensus_queue.cc:260] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.118575 12532 ts_tablet_manager.cc:1398] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.118853 12534 tablet_bootstrap.cc:654] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.119871 12534 tablet_bootstrap.cc:492] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:24.120000 12534 ts_tablet_manager.cc:1367] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:24.120379 12534 raft_consensus.cc:348] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.120469 12534 raft_consensus.cc:374] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.120496 12534 raft_consensus.cc:729] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.120620 12534 consensus_queue.cc:260] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.120882 12534 ts_tablet_manager.cc:1398] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:24.147495 12443 raft_consensus.cc:482] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:24.147629 12443 raft_consensus.cc:504] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.148015 12443 leader_election.cc:288] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:24.148396 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858f60ea9d814c69ac626cd22c9ea994" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:24.148417 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858f60ea9d814c69ac626cd22c9ea994" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:24.148535 11511 raft_consensus.cc:2454] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:24.148566 11380 raft_consensus.cc:2454] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:24.148831 11577 leader_election.cc:302] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:24.149003 12443 raft_consensus.cc:2781] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:24.149061 12443 raft_consensus.cc:482] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:24.149086 12443 raft_consensus.cc:3037] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.149847 12469 raft_consensus.cc:482] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:24.149942 12469 raft_consensus.cc:504] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.150151 12443 raft_consensus.cc:504] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.150300 12469 leader_election.cc:288] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:24.150497 12443 leader_election.cc:288] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:24.150662 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c86f45702e94c0e898e4e81148f2098" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:24.150796 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858f60ea9d814c69ac626cd22c9ea994" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:24.150796 11248 raft_consensus.cc:2454] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:24.150817 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858f60ea9d814c69ac626cd22c9ea994" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:24.150895 11511 raft_consensus.cc:3037] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.150935 11380 raft_consensus.cc:3037] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.151186 11445 leader_election.cc:302] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:24.151348 12469 raft_consensus.cc:2781] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:24.151389 12469 raft_consensus.cc:482] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:24.151420 12469 raft_consensus.cc:3037] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.151896 11511 raft_consensus.cc:2454] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:24.151896 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c86f45702e94c0e898e4e81148f2098" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:24.151973 11380 raft_consensus.cc:2454] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:24.152035 11381 raft_consensus.cc:2454] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:24.152151 11577 leader_election.cc:302] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:24.152575 12443 raft_consensus.cc:2781] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:24.152666 12443 raft_consensus.cc:686] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:24.152806 12443 consensus_queue.cc:237] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.154337 11098 catalog_manager.cc:5484] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:24.172346 12469 raft_consensus.cc:504] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.172801 12469 leader_election.cc:288] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:24.173238 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c86f45702e94c0e898e4e81148f2098" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:24.173285 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c86f45702e94c0e898e4e81148f2098" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:24.173388 11248 raft_consensus.cc:3037] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.173405 11381 raft_consensus.cc:3037] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.174401 11248 raft_consensus.cc:2454] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:24.174687 11381 raft_consensus.cc:2454] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:24.174751 11445 leader_election.cc:302] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:24.174913 12469 raft_consensus.cc:2781] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:24.175000 12469 raft_consensus.cc:686] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:24.175166 12469 consensus_queue.cc:237] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.176889 11098 catalog_manager.cc:5484] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:24.295733 12373 raft_consensus.cc:482] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:24.295853 12373 raft_consensus.cc:504] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.296274 12373 leader_election.cc:288] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:24.296648 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a10926bce14cf1bd12f21287e08c89" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:24.296671 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a10926bce14cf1bd12f21287e08c89" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:24.296797 11511 raft_consensus.cc:2454] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:24.296795 11643 raft_consensus.cc:2454] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:24.297098 11313 leader_election.cc:302] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:24.297309 12373 raft_consensus.cc:2781] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:24.297372 12373 raft_consensus.cc:482] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:24.297410 12373 raft_consensus.cc:3037] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.298242 12373 raft_consensus.cc:504] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.298576 12373 leader_election.cc:288] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:24.298921 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a10926bce14cf1bd12f21287e08c89" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:24.298933 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a10926bce14cf1bd12f21287e08c89" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:24.299041 11511 raft_consensus.cc:3037] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.299041 11643 raft_consensus.cc:3037] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.299796 11643 raft_consensus.cc:2454] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:24.299806 11511 raft_consensus.cc:2454] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:24.300100 11313 leader_election.cc:302] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:24.300289 12373 raft_consensus.cc:2781] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:24.300400 12373 raft_consensus.cc:686] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:24.300545 12373 consensus_queue.cc:237] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.302325 11098 catalog_manager.cc:5484] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:24.454591 12599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:24.454839 12599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:24.457958 12599 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 19:56:24.547741 11381 raft_consensus.cc:1261] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.548172 12443 consensus_queue.cc:1035] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.549589 11511 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.549932 11643 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.550101 12373 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.550324 12597 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.555372 11512 raft_consensus.cc:1261] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.556514 12582 consensus_queue.cc:1035] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.571533 12514 mvcc.cc:205] Tried to move back new op lower bound from 7022974503110365184 to 7022974502096519168. Current Snapshot: MvccSnapshot[applied={T|T < 7022974503110365184}]
I20240501 19:56:24.573150 12519 mvcc.cc:205] Tried to move back new op lower bound from 7022974503101423616 to 7022974501491130368. Current Snapshot: MvccSnapshot[applied={T|T < 7022974503101423616 or (T in {7022974503101423616})}]
I20240501 19:56:24.584481 11380 raft_consensus.cc:1261] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.584898 12469 consensus_queue.cc:1035] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.585224 11248 raft_consensus.cc:1261] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:24.586380 12243 consensus_queue.cc:1035] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:24.601923 12638 mvcc.cc:205] Tried to move back new op lower bound from 7022974503239213056 to 7022974501583273984. Current Snapshot: MvccSnapshot[applied={T|T < 7022974503239213056}]
INSERT report
rows total: 2000
time total: 124.747 ms
time per row: 0.0623736 ms
I20240501 19:56:24.623776 11101 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:24.629055 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 1c43596467ca48a58c42ff487b3fce90 (DEFAULT_TABLE table=rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.629133 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 1c43596467ca48a58c42ff487b3fce90 (DEFAULT_TABLE table=rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.629384 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c43596467ca48a58c42ff487b3fce90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.629429 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c43596467ca48a58c42ff487b3fce90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.629449 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 1c43596467ca48a58c42ff487b3fce90 (DEFAULT_TABLE table=rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.629698 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c43596467ca48a58c42ff487b3fce90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.630144 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2e2992240f49bdb3e81e5516199277 (DEFAULT_TABLE table=rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.630376 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2e2992240f49bdb3e81e5516199277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.630533 11492 tablet_service.cc:1467] Processing CreateTablet for tablet df804ea6956f4c15a8990e3a12945ff8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.630776 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df804ea6956f4c15a8990e3a12945ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.631884 11624 tablet_service.cc:1467] Processing CreateTablet for tablet df804ea6956f4c15a8990e3a12945ff8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.632113 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df804ea6956f4c15a8990e3a12945ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.632217 11229 tablet_service.cc:1467] Processing CreateTablet for tablet df804ea6956f4c15a8990e3a12945ff8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.632464 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df804ea6956f4c15a8990e3a12945ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.632555 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2e2992240f49bdb3e81e5516199277 (DEFAULT_TABLE table=rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.632766 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2e2992240f49bdb3e81e5516199277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.633460 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2e2992240f49bdb3e81e5516199277 (DEFAULT_TABLE table=rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:24.633512 12652 tablet_bootstrap.cc:492] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:24.633675 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2e2992240f49bdb3e81e5516199277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.634007 12650 tablet_bootstrap.cc:492] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:24.634594 12652 tablet_bootstrap.cc:654] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.634807 12650 tablet_bootstrap.cc:654] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.635430 12651 tablet_bootstrap.cc:492] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:24.635739 12652 tablet_bootstrap.cc:492] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:24.635821 12652 ts_tablet_manager.cc:1367] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:24.636260 12652 raft_consensus.cc:348] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.636389 12652 raft_consensus.cc:374] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.636426 12652 raft_consensus.cc:729] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.636453 12650 tablet_bootstrap.cc:492] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:24.636528 12650 ts_tablet_manager.cc:1367] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:24.636554 12651 tablet_bootstrap.cc:654] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.636570 12652 consensus_queue.cc:260] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.636991 12650 raft_consensus.cc:348] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.637094 12650 raft_consensus.cc:374] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.637125 12650 raft_consensus.cc:729] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.637259 12650 consensus_queue.cc:260] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.637313 12652 ts_tablet_manager.cc:1398] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.637699 12650 ts_tablet_manager.cc:1398] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.638235 12651 tablet_bootstrap.cc:492] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:24.638372 12651 ts_tablet_manager.cc:1367] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:24.638881 12651 raft_consensus.cc:348] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.639046 12651 raft_consensus.cc:374] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.639122 12651 raft_consensus.cc:729] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.639262 12651 consensus_queue.cc:260] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.639685 12651 ts_tablet_manager.cc:1398] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.639856 12651 tablet_bootstrap.cc:492] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:24.638995 12650 tablet_bootstrap.cc:492] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:24.640738 12651 tablet_bootstrap.cc:654] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.640960 12650 tablet_bootstrap.cc:654] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.641809 12651 tablet_bootstrap.cc:492] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:24.641896 12651 ts_tablet_manager.cc:1367] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:24.642053 12650 tablet_bootstrap.cc:492] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:24.642179 12650 ts_tablet_manager.cc:1367] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:24.642376 12651 raft_consensus.cc:348] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:24.642484 12651 raft_consensus.cc:374] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.642537 12651 raft_consensus.cc:729] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.642663 12651 consensus_queue.cc:260] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:24.642657 12650 raft_consensus.cc:348] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.642750 12650 raft_consensus.cc:374] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.642786 12650 raft_consensus.cc:729] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.642906 12651 ts_tablet_manager.cc:1398] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:24.642910 12650 consensus_queue.cc:260] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.643213 12650 ts_tablet_manager.cc:1398] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.643512 12650 tablet_bootstrap.cc:492] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:24.644594 12650 tablet_bootstrap.cc:654] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.645659 12650 tablet_bootstrap.cc:492] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:24.645737 12650 ts_tablet_manager.cc:1367] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:24.646116 12650 raft_consensus.cc:348] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:24.646238 12650 raft_consensus.cc:374] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.646277 12650 raft_consensus.cc:729] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.646409 12650 consensus_queue.cc:260] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:24.646651 12650 ts_tablet_manager.cc:1398] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:24.649519 12659 tablet_bootstrap.cc:492] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:24.650255 12659 tablet_bootstrap.cc:654] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.651183 12659 tablet_bootstrap.cc:492] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:24.651258 12659 ts_tablet_manager.cc:1367] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:24.651614 12659 raft_consensus.cc:348] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:24.651721 12659 raft_consensus.cc:374] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.651759 12659 raft_consensus.cc:729] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.651881 12659 consensus_queue.cc:260] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:24.652166 12659 ts_tablet_manager.cc:1398] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.652345 12659 tablet_bootstrap.cc:492] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:24.653158 12659 tablet_bootstrap.cc:654] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.654098 12659 tablet_bootstrap.cc:492] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:24.654179 12659 ts_tablet_manager.cc:1367] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:24.654583 12659 raft_consensus.cc:348] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.654685 12659 raft_consensus.cc:374] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.654721 12659 raft_consensus.cc:729] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.654829 12659 consensus_queue.cc:260] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:24.655086 12659 ts_tablet_manager.cc:1398] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:24.655230 12659 tablet_bootstrap.cc:492] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:24.656414 12659 tablet_bootstrap.cc:654] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:24.657316 12659 tablet_bootstrap.cc:492] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:24.657382 12659 ts_tablet_manager.cc:1367] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:24.657724 12659 raft_consensus.cc:348] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.657801 12659 raft_consensus.cc:374] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:24.657820 12659 raft_consensus.cc:729] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:24.657923 12659 consensus_queue.cc:260] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:24.658187 12659 ts_tablet_manager.cc:1398] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:24.772260 12585 raft_consensus.cc:482] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:24.772410 12585 raft_consensus.cc:504] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:24.772954 12585 leader_election.cc:288] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:24.773377 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e2992240f49bdb3e81e5516199277" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:24.773433 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e2992240f49bdb3e81e5516199277" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:24.773535 11643 raft_consensus.cc:2454] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:24.773552 11380 raft_consensus.cc:2454] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:24.773842 11183 leader_election.cc:302] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:24.774120 12585 raft_consensus.cc:2781] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:24.774194 12585 raft_consensus.cc:482] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:24.774235 12585 raft_consensus.cc:3037] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.775113 12585 raft_consensus.cc:504] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:24.775485 12585 leader_election.cc:288] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:24.775801 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e2992240f49bdb3e81e5516199277" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:24.775844 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e2992240f49bdb3e81e5516199277" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:24.775923 11643 raft_consensus.cc:3037] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.775966 11380 raft_consensus.cc:3037] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:24.776700 11643 raft_consensus.cc:2454] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:24.776707 11380 raft_consensus.cc:2454] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:24.776983 11182 leader_election.cc:302] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:24.777143 12585 raft_consensus.cc:2781] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:24.777251 12585 raft_consensus.cc:686] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:24.777400 12585 consensus_queue.cc:237] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:24.779250 11101 catalog_manager.cc:5484] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:25.024466 12646 raft_consensus.cc:482] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:25.024606 12646 raft_consensus.cc:504] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.025027 12646 leader_election.cc:288] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:25.025485 12585 raft_consensus.cc:482] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:25.025502 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df804ea6956f4c15a8990e3a12945ff8" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:25.025557 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df804ea6956f4c15a8990e3a12945ff8" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:25.025583 12585 raft_consensus.cc:504] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.025677 11643 raft_consensus.cc:2454] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:25.025705 11248 raft_consensus.cc:2454] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:25.025995 11446 leader_election.cc:302] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:25.026170 12646 raft_consensus.cc:2781] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:25.026257 12646 raft_consensus.cc:482] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:25.026304 12646 raft_consensus.cc:3037] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.026458 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c43596467ca48a58c42ff487b3fce90" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:25.026597 11380 raft_consensus.cc:2454] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:25.026886 11183 leader_election.cc:302] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:25.027052 12585 leader_election.cc:288] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:25.027140 12585 raft_consensus.cc:2781] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:25.027176 12585 raft_consensus.cc:482] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:25.027137 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c43596467ca48a58c42ff487b3fce90" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:25.027209 12585 raft_consensus.cc:3037] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.027289 11643 raft_consensus.cc:2454] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:25.027478 12646 raft_consensus.cc:504] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.027889 12646 leader_election.cc:288] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:25.028227 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df804ea6956f4c15a8990e3a12945ff8" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:25.028260 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df804ea6956f4c15a8990e3a12945ff8" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:25.028345 11643 raft_consensus.cc:3037] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.028363 11248 raft_consensus.cc:3037] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.028326 12585 raft_consensus.cc:504] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.028937 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c43596467ca48a58c42ff487b3fce90" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:25.029034 11644 raft_consensus.cc:3037] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.029014 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c43596467ca48a58c42ff487b3fce90" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:25.029138 11381 raft_consensus.cc:3037] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.029219 11248 raft_consensus.cc:2454] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:25.029372 11643 raft_consensus.cc:2454] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:25.029671 11446 leader_election.cc:302] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:25.029853 12646 raft_consensus.cc:2781] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:25.029911 11644 raft_consensus.cc:2454] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:25.029955 12646 raft_consensus.cc:686] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:25.030124 11381 raft_consensus.cc:2454] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:25.030112 12646 consensus_queue.cc:237] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.030288 12585 leader_election.cc:288] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:25.030329 11182 leader_election.cc:302] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:25.030483 12585 raft_consensus.cc:2781] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:25.030565 12585 raft_consensus.cc:686] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:25.030678 12585 consensus_queue.cc:237] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.032068 11101 catalog_manager.cc:5484] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.033051 11097 catalog_manager.cc:5484] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:25.191128 12666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:25.191465 12666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:25.195729 12666 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 19:56:25.253918 12585 consensus_queue.cc:1035] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.262518 12451 consensus_queue.cc:1035] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.280812 11644 raft_consensus.cc:1261] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.280834 11381 raft_consensus.cc:1261] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.281983 12585 consensus_queue.cc:1035] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.282306 12585 consensus_queue.cc:1035] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.292562 11644 raft_consensus.cc:1261] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.292950 12646 consensus_queue.cc:1035] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.298135 11248 raft_consensus.cc:1261] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.298503 12646 consensus_queue.cc:1035] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.22 ms
time per row: 0.0531101 ms
I20240501 19:56:25.343815 11097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:25.349447 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3176bda2ad4b9986cda48e4a03ce49 (DEFAULT_TABLE table=rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.349540 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3176bda2ad4b9986cda48e4a03ce49 (DEFAULT_TABLE table=rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.349778 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3176bda2ad4b9986cda48e4a03ce49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.349778 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3176bda2ad4b9986cda48e4a03ce49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.350438 11490 tablet_service.cc:1467] Processing CreateTablet for tablet 5e63472cfff449089d88d18b9e495912 (DEFAULT_TABLE table=rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.350745 11490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e63472cfff449089d88d18b9e495912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.350708 11623 tablet_service.cc:1467] Processing CreateTablet for tablet e96e8e7c66584b1198bea464e5a1a4d8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.350960 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96e8e7c66584b1198bea464e5a1a4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.351706 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3176bda2ad4b9986cda48e4a03ce49 (DEFAULT_TABLE table=rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.351931 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3176bda2ad4b9986cda48e4a03ce49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.353217 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 5e63472cfff449089d88d18b9e495912 (DEFAULT_TABLE table=rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.353289 11228 tablet_service.cc:1467] Processing CreateTablet for tablet e96e8e7c66584b1198bea464e5a1a4d8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.353436 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e63472cfff449089d88d18b9e495912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.353519 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96e8e7c66584b1198bea464e5a1a4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.354974 12721 tablet_bootstrap.cc:492] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:25.350438 11491 tablet_service.cc:1467] Processing CreateTablet for tablet e96e8e7c66584b1198bea464e5a1a4d8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.355484 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96e8e7c66584b1198bea464e5a1a4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.355986 12721 tablet_bootstrap.cc:654] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.355944 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 5e63472cfff449089d88d18b9e495912 (DEFAULT_TABLE table=rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.356161 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e63472cfff449089d88d18b9e495912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.356631 12724 tablet_bootstrap.cc:492] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:25.357553 12724 tablet_bootstrap.cc:654] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.358273 12723 tablet_bootstrap.cc:492] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:25.359134 12724 tablet_bootstrap.cc:492] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:25.359237 12724 ts_tablet_manager.cc:1367] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:25.359405 12722 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:25.359789 12723 tablet_bootstrap.cc:654] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.359899 12724 raft_consensus.cc:348] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.360018 12724 raft_consensus.cc:374] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.360059 12724 raft_consensus.cc:729] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.360209 12724 consensus_queue.cc:260] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.360597 12724 ts_tablet_manager.cc:1398] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.360769 12724 tablet_bootstrap.cc:492] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:25.361230 12722 tablet_bootstrap.cc:654] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.362262 12724 tablet_bootstrap.cc:654] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.362586 12723 tablet_bootstrap.cc:492] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:25.362668 12723 ts_tablet_manager.cc:1367] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:25.363327 12722 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:25.363409 12722 ts_tablet_manager.cc:1367] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:25.363826 12722 raft_consensus.cc:348] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.363920 12722 raft_consensus.cc:374] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.363950 12722 raft_consensus.cc:729] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.364254 12723 raft_consensus.cc:348] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:25.364346 12723 raft_consensus.cc:374] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.364373 12723 raft_consensus.cc:729] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.364532 12723 consensus_queue.cc:260] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:25.364823 12723 ts_tablet_manager.cc:1398] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:25.364997 12722 consensus_queue.cc:260] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.365300 12722 ts_tablet_manager.cc:1398] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:25.365768 12723 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:25.366667 12723 tablet_bootstrap.cc:654] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.367239 12722 tablet_bootstrap.cc:492] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:25.369235 12722 tablet_bootstrap.cc:654] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.370550 12722 tablet_bootstrap.cc:492] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:25.370632 12724 tablet_bootstrap.cc:492] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:25.370672 12722 ts_tablet_manager.cc:1367] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:25.370738 12724 ts_tablet_manager.cc:1367] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:56:25.370836 12721 tablet_bootstrap.cc:492] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:25.370934 12721 ts_tablet_manager.cc:1367] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:56:25.371229 12724 raft_consensus.cc:348] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:25.371346 12724 raft_consensus.cc:374] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.371387 12724 raft_consensus.cc:729] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.371359 12721 raft_consensus.cc:348] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.371484 12721 raft_consensus.cc:374] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.371526 12721 raft_consensus.cc:729] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.371254 12722 raft_consensus.cc:348] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.371573 12724 consensus_queue.cc:260] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:25.371616 12722 raft_consensus.cc:374] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.371654 12721 consensus_queue.cc:260] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.371764 12722 raft_consensus.cc:729] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.371862 12724 ts_tablet_manager.cc:1398] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.371906 12721 ts_tablet_manager.cc:1398] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.372004 12722 consensus_queue.cc:260] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.372084 12721 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:25.373152 12721 tablet_bootstrap.cc:654] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.374418 12723 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:25.374516 12723 ts_tablet_manager.cc:1367] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:56:25.374547 12721 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:25.374619 12721 ts_tablet_manager.cc:1367] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:25.374964 12723 raft_consensus.cc:348] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.375083 12723 raft_consensus.cc:374] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.375118 12723 raft_consensus.cc:729] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.375219 12721 raft_consensus.cc:348] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.375248 12723 consensus_queue.cc:260] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.375317 12721 raft_consensus.cc:374] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.375351 12721 raft_consensus.cc:729] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.375505 12721 consensus_queue.cc:260] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.375881 12722 ts_tablet_manager.cc:1398] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:25.376245 12723 ts_tablet_manager.cc:1398] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:25.376488 12721 ts_tablet_manager.cc:1398] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:25.376670 12722 tablet_bootstrap.cc:492] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:25.377506 12722 tablet_bootstrap.cc:654] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.380537 12722 tablet_bootstrap.cc:492] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:25.380620 12722 ts_tablet_manager.cc:1367] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:25.380995 12722 raft_consensus.cc:348] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:25.381094 12722 raft_consensus.cc:374] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.381129 12722 raft_consensus.cc:729] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.381238 12722 consensus_queue.cc:260] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:25.381515 12722 ts_tablet_manager.cc:1398] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:25.444350 12585 raft_consensus.cc:482] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:25.444494 12585 raft_consensus.cc:504] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.444988 12585 leader_election.cc:288] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:25.445464 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:25.445552 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:25.445626 11380 raft_consensus.cc:2454] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:25.445731 11512 raft_consensus.cc:2454] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:25.446007 11183 leader_election.cc:302] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:25.446173 12585 raft_consensus.cc:2781] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:25.446244 12585 raft_consensus.cc:482] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:25.446282 12585 raft_consensus.cc:3037] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.447078 12585 raft_consensus.cc:504] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.447378 12585 leader_election.cc:288] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:25.447713 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:25.447752 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:25.447824 11380 raft_consensus.cc:3037] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.447844 11512 raft_consensus.cc:3037] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.448554 11380 raft_consensus.cc:2454] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:25.448606 11512 raft_consensus.cc:2454] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:25.448848 11183 leader_election.cc:302] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:25.449034 12585 raft_consensus.cc:2781] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:25.449115 12585 raft_consensus.cc:686] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:25.449267 12585 consensus_queue.cc:237] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.449564 12451 raft_consensus.cc:482] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:25.449646 12451 raft_consensus.cc:504] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:25.449970 12451 leader_election.cc:288] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:25.450255 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96e8e7c66584b1198bea464e5a1a4d8" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:25.450379 11512 raft_consensus.cc:2454] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:25.450548 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96e8e7c66584b1198bea464e5a1a4d8" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:25.450661 11181 leader_election.cc:302] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:25.450678 11644 raft_consensus.cc:2454] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:25.450814 12451 raft_consensus.cc:2781] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:25.450860 12451 raft_consensus.cc:482] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:25.450894 12451 raft_consensus.cc:3037] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.451174 11097 catalog_manager.cc:5484] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:25.451685 12451 raft_consensus.cc:504] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:25.452028 12451 leader_election.cc:288] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:25.452359 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96e8e7c66584b1198bea464e5a1a4d8" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:25.452474 11644 raft_consensus.cc:3037] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.452606 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96e8e7c66584b1198bea464e5a1a4d8" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:25.452708 11512 raft_consensus.cc:3037] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.453439 11644 raft_consensus.cc:2454] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:25.453519 11512 raft_consensus.cc:2454] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:25.453727 11182 leader_election.cc:302] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:25.453882 12451 raft_consensus.cc:2781] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:25.453961 12451 raft_consensus.cc:686] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:25.454113 12451 consensus_queue.cc:237] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:25.455796 11097 catalog_manager.cc:5484] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.491658 12469 raft_consensus.cc:482] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:25.491791 12469 raft_consensus.cc:504] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.492121 12469 leader_election.cc:288] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:25.492491 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e63472cfff449089d88d18b9e495912" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:25.492499 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e63472cfff449089d88d18b9e495912" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:25.492614 11380 raft_consensus.cc:2454] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:25.492643 11644 raft_consensus.cc:2454] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:25.492903 11446 leader_election.cc:302] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:25.493073 12469 raft_consensus.cc:2781] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:25.493129 12469 raft_consensus.cc:482] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:25.493155 12469 raft_consensus.cc:3037] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.493911 12469 raft_consensus.cc:504] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.494217 12469 leader_election.cc:288] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:25.494511 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e63472cfff449089d88d18b9e495912" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:25.494500 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e63472cfff449089d88d18b9e495912" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:25.494626 11380 raft_consensus.cc:3037] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.494626 11644 raft_consensus.cc:3037] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.495543 11380 raft_consensus.cc:2454] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:25.495543 11644 raft_consensus.cc:2454] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:25.495810 11447 leader_election.cc:302] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:25.495955 12469 raft_consensus.cc:2781] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:25.496037 12469 raft_consensus.cc:686] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:25.496188 12469 consensus_queue.cc:237] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.497918 11097 catalog_manager.cc:5484] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:25.618100 12737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:25.618376 12737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:25.621582 12737 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 19:56:25.701895 11380 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.702308 12646 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.704257 11644 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.704886 11512 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.705065 11381 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.705147 12646 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.705559 12585 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.705849 12585 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.711669 11643 raft_consensus.cc:1261] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.711786 11511 raft_consensus.cc:1261] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:25.712090 12585 consensus_queue.cc:1035] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.715811 12451 consensus_queue.cc:1035] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.303 ms
time per row: 0.0566517 ms
I20240501 19:56:25.777532 11098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:25.783422 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0c6b4db2cc467da100045058e7b52b (DEFAULT_TABLE table=rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.783793 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0c6b4db2cc467da100045058e7b52b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.783725 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0c6b4db2cc467da100045058e7b52b (DEFAULT_TABLE table=rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.784013 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0c6b4db2cc467da100045058e7b52b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.784361 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 3d99f4d3127348dba949ab5fd4666956 (DEFAULT_TABLE table=rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.784628 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d99f4d3127348dba949ab5fd4666956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.785256 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 3d99f4d3127348dba949ab5fd4666956 (DEFAULT_TABLE table=rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.785504 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d99f4d3127348dba949ab5fd4666956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.786094 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 03cf15fd86404aaab8e93953b06a22cc (DEFAULT_TABLE table=rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.786165 12721 tablet_bootstrap.cc:492] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:25.786310 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03cf15fd86404aaab8e93953b06a22cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.786661 12724 tablet_bootstrap.cc:492] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:25.787205 12721 tablet_bootstrap.cc:654] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.787673 12724 tablet_bootstrap.cc:654] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.787881 12722 tablet_bootstrap.cc:492] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:25.788102 12723 tablet_bootstrap.cc:492] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:25.788344 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0c6b4db2cc467da100045058e7b52b (DEFAULT_TABLE table=rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.788527 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0c6b4db2cc467da100045058e7b52b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.788666 12722 tablet_bootstrap.cc:654] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.788889 12723 tablet_bootstrap.cc:654] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.789690 12722 tablet_bootstrap.cc:492] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:25.789775 12722 ts_tablet_manager.cc:1367] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:25.790184 12723 tablet_bootstrap.cc:492] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:25.790266 12723 ts_tablet_manager.cc:1367] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:25.790206 12722 raft_consensus.cc:348] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.790318 12722 raft_consensus.cc:374] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.790354 12722 raft_consensus.cc:729] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.790478 12722 consensus_queue.cc:260] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.790622 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 03cf15fd86404aaab8e93953b06a22cc (DEFAULT_TABLE table=rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.790688 12723 raft_consensus.cc:348] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.790757 12722 ts_tablet_manager.cc:1398] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.790791 12723 raft_consensus.cc:374] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.790832 12723 raft_consensus.cc:729] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.790864 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03cf15fd86404aaab8e93953b06a22cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.790912 12722 tablet_bootstrap.cc:492] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:25.790954 12723 consensus_queue.cc:260] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.791232 12723 ts_tablet_manager.cc:1398] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.791397 12723 tablet_bootstrap.cc:492] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:25.792210 12722 tablet_bootstrap.cc:654] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.792649 12723 tablet_bootstrap.cc:654] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.792749 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 3d99f4d3127348dba949ab5fd4666956 (DEFAULT_TABLE table=rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.793056 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d99f4d3127348dba949ab5fd4666956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.793771 12724 tablet_bootstrap.cc:492] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:25.793859 12724 ts_tablet_manager.cc:1367] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:25.794312 12724 raft_consensus.cc:348] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.794413 12724 raft_consensus.cc:374] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.794445 12724 raft_consensus.cc:729] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.794567 12724 consensus_queue.cc:260] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.794728 12722 tablet_bootstrap.cc:492] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:25.794822 12724 ts_tablet_manager.cc:1398] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.794966 12724 tablet_bootstrap.cc:492] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:25.795276 12722 ts_tablet_manager.cc:1367] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:56:25.795738 12722 raft_consensus.cc:348] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.795868 12722 raft_consensus.cc:374] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.795899 12722 raft_consensus.cc:729] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.796106 12723 tablet_bootstrap.cc:492] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:25.796197 12723 ts_tablet_manager.cc:1367] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:25.796608 12723 raft_consensus.cc:348] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:25.796710 12723 raft_consensus.cc:374] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.796742 12723 raft_consensus.cc:729] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.796808 12722 consensus_queue.cc:260] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.796849 12723 consensus_queue.cc:260] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:25.797106 12723 ts_tablet_manager.cc:1398] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.791580 12721 tablet_bootstrap.cc:492] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:25.797371 12721 ts_tablet_manager.cc:1367] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20240501 19:56:25.797827 12721 raft_consensus.cc:348] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.797931 12721 raft_consensus.cc:374] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.797961 12721 raft_consensus.cc:729] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.798064 12722 ts_tablet_manager.cc:1398] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:25.798066 12721 consensus_queue.cc:260] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.798288 12724 tablet_bootstrap.cc:654] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.799135 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 03cf15fd86404aaab8e93953b06a22cc (DEFAULT_TABLE table=rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:25.799387 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03cf15fd86404aaab8e93953b06a22cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.800279 12724 tablet_bootstrap.cc:492] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:25.800351 12724 ts_tablet_manager.cc:1367] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:25.800741 12724 raft_consensus.cc:348] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:25.800828 12724 raft_consensus.cc:374] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.800848 12724 raft_consensus.cc:729] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.800958 12724 consensus_queue.cc:260] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:25.801441 12721 ts_tablet_manager.cc:1398] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:25.801656 12721 tablet_bootstrap.cc:492] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:25.801666 12724 ts_tablet_manager.cc:1398] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.801826 12724 tablet_bootstrap.cc:492] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:25.802420 12721 tablet_bootstrap.cc:654] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.802646 12724 tablet_bootstrap.cc:654] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:25.803540 12721 tablet_bootstrap.cc:492] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:25.803632 12721 ts_tablet_manager.cc:1367] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:25.804051 12721 raft_consensus.cc:348] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:25.804155 12721 raft_consensus.cc:374] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.804191 12721 raft_consensus.cc:729] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.804312 12721 consensus_queue.cc:260] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:25.804373 12724 tablet_bootstrap.cc:492] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:25.804442 12724 ts_tablet_manager.cc:1367] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:25.804579 12721 ts_tablet_manager.cc:1398] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.804842 12724 raft_consensus.cc:348] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.804958 12724 raft_consensus.cc:374] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:25.804992 12724 raft_consensus.cc:729] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:25.805099 12724 consensus_queue.cc:260] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.805593 12724 ts_tablet_manager.cc:1398] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.835850 12643 raft_consensus.cc:482] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:25.835973 12643 raft_consensus.cc:504] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.836337 12643 leader_election.cc:288] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:25.836694 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d99f4d3127348dba949ab5fd4666956" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:25.836822 11248 raft_consensus.cc:2454] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:25.836944 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d99f4d3127348dba949ab5fd4666956" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:25.837076 11577 leader_election.cc:302] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:25.837103 11511 raft_consensus.cc:2454] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:25.837226 12643 raft_consensus.cc:2781] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:25.837280 12643 raft_consensus.cc:482] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:25.837318 12643 raft_consensus.cc:3037] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.838129 12643 raft_consensus.cc:504] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.838440 12643 leader_election.cc:288] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:25.838784 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d99f4d3127348dba949ab5fd4666956" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:25.838820 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d99f4d3127348dba949ab5fd4666956" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:25.838905 11248 raft_consensus.cc:3037] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.838918 11511 raft_consensus.cc:3037] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.839747 11248 raft_consensus.cc:2454] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:25.839818 11511 raft_consensus.cc:2454] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:25.840032 11577 leader_election.cc:302] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:25.840190 12643 raft_consensus.cc:2781] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:25.840391 12643 raft_consensus.cc:686] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:25.840535 12643 consensus_queue.cc:237] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:25.842330 11098 catalog_manager.cc:5484] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.905995 12694 raft_consensus.cc:482] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:25.906141 12694 raft_consensus.cc:504] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.906608 12694 leader_election.cc:288] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:25.906996 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0c6b4db2cc467da100045058e7b52b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:25.907055 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0c6b4db2cc467da100045058e7b52b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:25.907137 11248 raft_consensus.cc:2454] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:25.907210 11511 raft_consensus.cc:2454] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:25.907429 11313 leader_election.cc:302] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:25.907637 12694 raft_consensus.cc:2781] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:25.907696 12694 raft_consensus.cc:482] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:25.907739 12694 raft_consensus.cc:3037] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.908598 12694 raft_consensus.cc:504] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.908928 12694 leader_election.cc:288] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:25.909258 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0c6b4db2cc467da100045058e7b52b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:25.909284 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0c6b4db2cc467da100045058e7b52b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:25.909394 11511 raft_consensus.cc:3037] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.909395 11248 raft_consensus.cc:3037] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:25.910126 11511 raft_consensus.cc:2454] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:25.910127 11248 raft_consensus.cc:2454] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:25.910403 11313 leader_election.cc:302] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:25.910568 12694 raft_consensus.cc:2781] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:25.910749 12694 raft_consensus.cc:686] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:25.910895 12694 consensus_queue.cc:237] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:25.912535 11098 catalog_manager.cc:5484] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:26.098860 12694 raft_consensus.cc:482] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:26.098995 12694 raft_consensus.cc:504] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:26.099357 12694 leader_election.cc:288] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:26.099673 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cf15fd86404aaab8e93953b06a22cc" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:26.099721 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cf15fd86404aaab8e93953b06a22cc" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:26.099798 11248 raft_consensus.cc:2454] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:26.099861 11644 raft_consensus.cc:2454] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:26.100131 11314 leader_election.cc:302] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:26.100332 12694 raft_consensus.cc:2781] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:26.100374 12694 raft_consensus.cc:482] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:26.100409 12694 raft_consensus.cc:3037] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.101173 12694 raft_consensus.cc:504] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:26.101475 12694 leader_election.cc:288] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:26.101799 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cf15fd86404aaab8e93953b06a22cc" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:26.101799 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03cf15fd86404aaab8e93953b06a22cc" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:26.101913 11248 raft_consensus.cc:3037] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.101913 11644 raft_consensus.cc:3037] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.102794 11644 raft_consensus.cc:2454] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:26.102794 11248 raft_consensus.cc:2454] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:26.103080 11313 leader_election.cc:302] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:26.103322 12694 raft_consensus.cc:2781] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:26.103399 12694 raft_consensus.cc:686] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:26.103557 12694 consensus_queue.cc:237] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:26.105293 11098 catalog_manager.cc:5484] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:26.220425 12784 consensus_queue.cc:1035] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:26.224228 12791 consensus_queue.cc:1035] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:26.266408 12789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:26.266683 12789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:26.270066 12789 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 19:56:26.368669 11248 raft_consensus.cc:1261] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:26.369297 11511 raft_consensus.cc:1261] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:26.369562 12786 consensus_queue.cc:1035] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:26.370237 12786 consensus_queue.cc:1035] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:26.372537 11644 raft_consensus.cc:1261] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:26.372891 12786 consensus_queue.cc:1035] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:26.375290 11249 raft_consensus.cc:1261] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:26.377985 12786 consensus_queue.cc:1035] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.566 ms
time per row: 0.0637831 ms
I20240501 19:56:26.438879 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:26.444850 11360 tablet_service.cc:1467] Processing CreateTablet for tablet d505c415daa54005b8ed22bdc11ce135 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:26.444989 11624 tablet_service.cc:1467] Processing CreateTablet for tablet d505c415daa54005b8ed22bdc11ce135 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:26.445091 11491 tablet_service.cc:1467] Processing CreateTablet for tablet d505c415daa54005b8ed22bdc11ce135 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:26.445184 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d505c415daa54005b8ed22bdc11ce135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.445307 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d505c415daa54005b8ed22bdc11ce135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.445351 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d505c415daa54005b8ed22bdc11ce135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.445648 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc7da07b6c5414bb305264962d0f218 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:26.445845 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7da07b6c5414bb305264962d0f218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.448223 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6a547744514cd88d167a0787d8535d (DEFAULT_TABLE table=rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:26.448247 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6a547744514cd88d167a0787d8535d (DEFAULT_TABLE table=rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:26.448307 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6a547744514cd88d167a0787d8535d (DEFAULT_TABLE table=rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:26.448391 12844 tablet_bootstrap.cc:492] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:26.448463 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6a547744514cd88d167a0787d8535d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.448503 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6a547744514cd88d167a0787d8535d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.448503 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6a547744514cd88d167a0787d8535d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.449520 12844 tablet_bootstrap.cc:654] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:26.450130 12845 tablet_bootstrap.cc:492] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:26.450910 12845 tablet_bootstrap.cc:654] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:26.451098 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc7da07b6c5414bb305264962d0f218 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:26.451319 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7da07b6c5414bb305264962d0f218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.451472 12846 tablet_bootstrap.cc:492] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:26.452014 12844 tablet_bootstrap.cc:492] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:26.452045 12845 tablet_bootstrap.cc:492] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:26.452108 12844 ts_tablet_manager.cc:1367] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:26.452124 12845 ts_tablet_manager.cc:1367] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:26.452492 12846 tablet_bootstrap.cc:654] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:26.452571 12844 raft_consensus.cc:348] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.452574 12845 raft_consensus.cc:348] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.452670 12844 raft_consensus.cc:374] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:26.452672 12845 raft_consensus.cc:374] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:26.452702 12845 raft_consensus.cc:729] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:26.452702 12844 raft_consensus.cc:729] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:26.452832 12844 consensus_queue.cc:260] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.452912 12845 consensus_queue.cc:260] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.453115 12844 ts_tablet_manager.cc:1398] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:26.453140 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc7da07b6c5414bb305264962d0f218 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:26.453281 12844 tablet_bootstrap.cc:492] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:26.453472 12845 ts_tablet_manager.cc:1398] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:26.453840 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7da07b6c5414bb305264962d0f218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.454017 12845 tablet_bootstrap.cc:492] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:26.454120 12844 tablet_bootstrap.cc:654] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:26.454881 12845 tablet_bootstrap.cc:654] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:26.455122 12844 tablet_bootstrap.cc:492] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:26.455199 12844 ts_tablet_manager.cc:1367] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:26.455631 12844 raft_consensus.cc:348] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.455739 12844 raft_consensus.cc:374] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:26.455771 12844 raft_consensus.cc:729] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:26.455880 12844 consensus_queue.cc:260] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.456130 12844 ts_tablet_manager.cc:1398] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:26.456319 12845 tablet_bootstrap.cc:492] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:26.456395 12845 ts_tablet_manager.cc:1367] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:26.456763 12844 tablet_bootstrap.cc:492] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:26.456811 12845 raft_consensus.cc:348] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.456897 12845 raft_consensus.cc:374] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:26.456926 12845 raft_consensus.cc:729] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:26.457017 12845 consensus_queue.cc:260] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.457067 12848 tablet_bootstrap.cc:492] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:26.457255 12845 ts_tablet_manager.cc:1398] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:26.457405 12845 tablet_bootstrap.cc:492] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:26.457589 12844 tablet_bootstrap.cc:654] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:26.458212 12845 tablet_bootstrap.cc:654] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:26.458230 12848 tablet_bootstrap.cc:654] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:26.459116 12846 tablet_bootstrap.cc:492] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:26.459257 12846 ts_tablet_manager.cc:1367] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:26.459659 12845 tablet_bootstrap.cc:492] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:26.459755 12845 ts_tablet_manager.cc:1367] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:26.459801 12848 tablet_bootstrap.cc:492] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:26.459887 12848 ts_tablet_manager.cc:1367] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:26.460043 12846 raft_consensus.cc:348] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.460198 12846 raft_consensus.cc:374] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:26.460271 12846 raft_consensus.cc:729] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:26.460330 12845 raft_consensus.cc:348] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:26.460327 12848 raft_consensus.cc:348] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.460443 12845 raft_consensus.cc:374] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:26.460459 12848 raft_consensus.cc:374] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:26.460484 12845 raft_consensus.cc:729] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:26.460492 12848 raft_consensus.cc:729] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:26.460575 12846 consensus_queue.cc:260] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.460592 12845 consensus_queue.cc:260] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:26.460842 12845 ts_tablet_manager.cc:1398] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:26.460915 12846 ts_tablet_manager.cc:1398] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:26.461441 12846 tablet_bootstrap.cc:492] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:26.461525 12844 tablet_bootstrap.cc:492] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:26.461611 12844 ts_tablet_manager.cc:1367] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:26.462008 12844 raft_consensus.cc:348] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:26.462107 12844 raft_consensus.cc:374] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:26.462139 12844 raft_consensus.cc:729] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:26.462257 12844 consensus_queue.cc:260] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:26.462330 12846 tablet_bootstrap.cc:654] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:26.462471 12848 consensus_queue.cc:260] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.462527 12844 ts_tablet_manager.cc:1398] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:26.462765 12848 ts_tablet_manager.cc:1398] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:26.463806 12846 tablet_bootstrap.cc:492] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:26.463892 12846 ts_tablet_manager.cc:1367] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:26.464428 12846 raft_consensus.cc:348] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:26.464535 12846 raft_consensus.cc:374] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:26.464568 12846 raft_consensus.cc:729] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:26.464671 12846 consensus_queue.cc:260] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:26.465421 12846 ts_tablet_manager.cc:1398] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:26.511009 12784 raft_consensus.cc:482] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:26.511142 12784 raft_consensus.cc:504] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.511605 12784 leader_election.cc:288] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:26.512013 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d505c415daa54005b8ed22bdc11ce135" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:26.512039 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d505c415daa54005b8ed22bdc11ce135" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:26.512140 11512 raft_consensus.cc:2454] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:26.512187 11380 raft_consensus.cc:2454] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:26.512468 11577 leader_election.cc:302] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:26.512647 12784 raft_consensus.cc:2781] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:26.512707 12784 raft_consensus.cc:482] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:26.512734 12784 raft_consensus.cc:3037] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.513650 12784 raft_consensus.cc:504] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.513978 12784 leader_election.cc:288] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:26.514298 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d505c415daa54005b8ed22bdc11ce135" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:26.514403 11512 raft_consensus.cc:3037] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.514425 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d505c415daa54005b8ed22bdc11ce135" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:26.514551 11380 raft_consensus.cc:3037] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.515203 11512 raft_consensus.cc:2454] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:26.515532 11380 raft_consensus.cc:2454] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:26.515776 11577 leader_election.cc:302] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:26.515964 12784 raft_consensus.cc:2781] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:26.516080 12784 raft_consensus.cc:686] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:26.516225 12784 consensus_queue.cc:237] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.518043 11105 catalog_manager.cc:5484] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:26.583039 12774 raft_consensus.cc:482] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:26.583160 12774 raft_consensus.cc:504] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.583578 12774 leader_election.cc:288] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:26.583993 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6a547744514cd88d167a0787d8535d" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:26.584039 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6a547744514cd88d167a0787d8535d" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:26.584141 11380 raft_consensus.cc:2454] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:26.584154 11249 raft_consensus.cc:2454] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:26.584487 11447 leader_election.cc:302] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:26.584663 12774 raft_consensus.cc:2781] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:26.584722 12774 raft_consensus.cc:482] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:26.584764 12774 raft_consensus.cc:3037] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.585623 12774 raft_consensus.cc:504] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.585968 12774 leader_election.cc:288] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:26.586267 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6a547744514cd88d167a0787d8535d" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:26.586396 11249 raft_consensus.cc:3037] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.586417 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6a547744514cd88d167a0787d8535d" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:26.586570 11380 raft_consensus.cc:3037] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.587312 11249 raft_consensus.cc:2454] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:26.587430 11380 raft_consensus.cc:2454] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:26.587636 11445 leader_election.cc:302] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:26.587810 12774 raft_consensus.cc:2781] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:26.587998 12774 raft_consensus.cc:686] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:26.588135 12774 consensus_queue.cc:237] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:26.589839 11105 catalog_manager.cc:5484] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:26.778376 12774 raft_consensus.cc:482] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:26.778548 12774 raft_consensus.cc:504] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:26.778975 12774 leader_election.cc:288] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:26.779487 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7da07b6c5414bb305264962d0f218" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:26.779480 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7da07b6c5414bb305264962d0f218" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:26.779647 11380 raft_consensus.cc:2454] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:26.779647 11644 raft_consensus.cc:2454] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:26.779945 11446 leader_election.cc:302] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:26.780125 12774 raft_consensus.cc:2781] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:26.780187 12774 raft_consensus.cc:482] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:26.780215 12774 raft_consensus.cc:3037] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.781033 12774 raft_consensus.cc:504] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:26.781389 12774 leader_election.cc:288] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:26.781759 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7da07b6c5414bb305264962d0f218" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:26.781759 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7da07b6c5414bb305264962d0f218" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:26.781891 11380 raft_consensus.cc:3037] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.781891 11644 raft_consensus.cc:3037] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:26.782835 11644 raft_consensus.cc:2454] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:26.782835 11380 raft_consensus.cc:2454] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:26.783186 11446 leader_election.cc:302] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:26.783370 12774 raft_consensus.cc:2781] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:26.783522 12774 raft_consensus.cc:686] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:26.783676 12774 consensus_queue.cc:237] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:26.786792 11105 catalog_manager.cc:5484] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:26.931674 12861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:26.931929 12861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:26.935107 12861 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 19:56:26.979041 12858 consensus_queue.cc:1035] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:26.991176 12858 consensus_queue.cc:1035] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.021629 11381 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.022496 11644 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.022959 12858 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.023531 12858 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.032691 11381 raft_consensus.cc:1261] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.033138 12784 consensus_queue.cc:1035] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.035239 11512 raft_consensus.cc:1261] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.036887 12784 consensus_queue.cc:1035] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 101.862 ms
time per row: 0.0509309 ms
I20240501 19:56:27.085341 11098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:27.090507 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 6070b62336054ef4b0444f27844b9ee1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.090579 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 6070b62336054ef4b0444f27844b9ee1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.090833 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6070b62336054ef4b0444f27844b9ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.090808 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 6070b62336054ef4b0444f27844b9ee1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.090888 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6070b62336054ef4b0444f27844b9ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.091042 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6070b62336054ef4b0444f27844b9ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.091811 11358 tablet_service.cc:1467] Processing CreateTablet for tablet f0fb00c1acda4d98a90b12d877412045 (DEFAULT_TABLE table=rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.092038 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fb00c1acda4d98a90b12d877412045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.092939 11228 tablet_service.cc:1467] Processing CreateTablet for tablet c7cd411ab06242d7809d56cd138fd7a5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.093179 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cd411ab06242d7809d56cd138fd7a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.093390 11227 tablet_service.cc:1467] Processing CreateTablet for tablet f0fb00c1acda4d98a90b12d877412045 (DEFAULT_TABLE table=rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.093611 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fb00c1acda4d98a90b12d877412045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.094213 12923 tablet_bootstrap.cc:492] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:27.094291 11491 tablet_service.cc:1467] Processing CreateTablet for tablet c7cd411ab06242d7809d56cd138fd7a5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.094566 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cd411ab06242d7809d56cd138fd7a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.095398 12923 tablet_bootstrap.cc:654] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.096424 11623 tablet_service.cc:1467] Processing CreateTablet for tablet c7cd411ab06242d7809d56cd138fd7a5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.096462 11624 tablet_service.cc:1467] Processing CreateTablet for tablet f0fb00c1acda4d98a90b12d877412045 (DEFAULT_TABLE table=rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.096663 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cd411ab06242d7809d56cd138fd7a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.096792 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fb00c1acda4d98a90b12d877412045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.096791 12925 tablet_bootstrap.cc:492] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:27.097739 12925 tablet_bootstrap.cc:654] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.098517 12921 tablet_bootstrap.cc:492] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:27.099014 12925 tablet_bootstrap.cc:492] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:27.099095 12925 ts_tablet_manager.cc:1367] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:27.099536 12921 tablet_bootstrap.cc:654] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.099500 12925 raft_consensus.cc:348] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.099584 12925 raft_consensus.cc:374] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.099612 12925 raft_consensus.cc:729] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.099758 12925 consensus_queue.cc:260] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.100034 12925 ts_tablet_manager.cc:1398] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.100325 12922 tablet_bootstrap.cc:492] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:27.100461 12923 tablet_bootstrap.cc:492] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:27.100548 12923 ts_tablet_manager.cc:1367] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 19:56:27.100965 12923 raft_consensus.cc:348] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.101078 12923 raft_consensus.cc:374] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.101112 12923 raft_consensus.cc:729] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.101249 12922 tablet_bootstrap.cc:654] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.101246 12923 consensus_queue.cc:260] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.102069 12923 ts_tablet_manager.cc:1398] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:27.102234 12923 tablet_bootstrap.cc:492] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:27.102499 12922 tablet_bootstrap.cc:492] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:27.102603 12922 ts_tablet_manager.cc:1367] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:27.102995 12923 tablet_bootstrap.cc:654] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.103049 12922 raft_consensus.cc:348] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:27.103160 12922 raft_consensus.cc:374] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.103152 12921 tablet_bootstrap.cc:492] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:27.103225 12922 raft_consensus.cc:729] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.103298 12921 ts_tablet_manager.cc:1367] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:56:27.103391 12922 consensus_queue.cc:260] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:27.103720 12922 ts_tablet_manager.cc:1398] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.104235 12922 tablet_bootstrap.cc:492] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:27.105142 12922 tablet_bootstrap.cc:654] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.105710 12921 raft_consensus.cc:348] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:27.105859 12923 tablet_bootstrap.cc:492] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:27.105891 12921 raft_consensus.cc:374] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.105952 12923 ts_tablet_manager.cc:1367] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:27.106021 12921 raft_consensus.cc:729] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.106216 12921 consensus_queue.cc:260] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:27.106350 12923 raft_consensus.cc:348] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:27.106483 12923 raft_consensus.cc:374] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.106528 12923 raft_consensus.cc:729] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.106664 12922 tablet_bootstrap.cc:492] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:27.106673 12923 consensus_queue.cc:260] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:27.106747 12922 ts_tablet_manager.cc:1367] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:27.106868 12921 ts_tablet_manager.cc:1398] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:27.107057 12921 tablet_bootstrap.cc:492] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:27.107158 12922 raft_consensus.cc:348] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.107252 12922 raft_consensus.cc:374] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.107286 12922 raft_consensus.cc:729] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.107419 12922 consensus_queue.cc:260] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.107713 12922 ts_tablet_manager.cc:1398] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.107856 12922 tablet_bootstrap.cc:492] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:27.108047 12921 tablet_bootstrap.cc:654] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.108652 12922 tablet_bootstrap.cc:654] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.109388 12921 tablet_bootstrap.cc:492] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:27.109414 12923 ts_tablet_manager.cc:1398] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:27.109459 12921 ts_tablet_manager.cc:1367] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:27.109848 12921 raft_consensus.cc:348] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.109949 12921 raft_consensus.cc:374] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.109982 12921 raft_consensus.cc:729] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.110090 12921 consensus_queue.cc:260] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.110348 12921 ts_tablet_manager.cc:1398] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.110493 12921 tablet_bootstrap.cc:492] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:27.110502 12922 tablet_bootstrap.cc:492] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:27.110564 12922 ts_tablet_manager.cc:1367] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:27.110934 12922 raft_consensus.cc:348] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.111033 12922 raft_consensus.cc:374] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.111061 12922 raft_consensus.cc:729] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.111163 12922 consensus_queue.cc:260] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.111316 12921 tablet_bootstrap.cc:654] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.111475 12922 ts_tablet_manager.cc:1398] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.112475 12921 tablet_bootstrap.cc:492] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:27.112545 12921 ts_tablet_manager.cc:1367] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:27.112881 12921 raft_consensus.cc:348] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.112977 12921 raft_consensus.cc:374] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.113006 12921 raft_consensus.cc:729] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.113128 12921 consensus_queue.cc:260] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.113360 12921 ts_tablet_manager.cc:1398] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:27.249532 12585 raft_consensus.cc:482] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:27.249660 12585 raft_consensus.cc:504] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.249847 12920 raft_consensus.cc:482] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:27.249966 12920 raft_consensus.cc:504] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.250110 12585 leader_election.cc:288] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:27.250347 12920 leader_election.cc:288] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:27.250464 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb00c1acda4d98a90b12d877412045" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:27.250594 11644 raft_consensus.cc:2454] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:27.250741 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb00c1acda4d98a90b12d877412045" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:27.250782 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb00c1acda4d98a90b12d877412045" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:27.250902 11381 raft_consensus.cc:2454] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:27.250918 11249 raft_consensus.cc:2454] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:27.250964 11182 leader_election.cc:302] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:27.250758 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb00c1acda4d98a90b12d877412045" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:27.251137 11380 raft_consensus.cc:2454] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:27.251207 12585 raft_consensus.cc:2781] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:27.251214 11577 leader_election.cc:302] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:27.251283 12585 raft_consensus.cc:482] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:27.251327 12585 raft_consensus.cc:3037] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.251379 12920 raft_consensus.cc:2781] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:27.251428 12920 raft_consensus.cc:482] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:27.251507 12920 raft_consensus.cc:3037] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.252394 12585 raft_consensus.cc:504] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.252394 12920 raft_consensus.cc:504] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.252723 12585 leader_election.cc:288] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:27.252974 12920 leader_election.cc:288] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:27.253033 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb00c1acda4d98a90b12d877412045" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:27.253032 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb00c1acda4d98a90b12d877412045" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:27.253178 11249 raft_consensus.cc:2379] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ad18cd54c6340fd92b874157875fa49 in current term 1: Already voted for candidate 10ee60ead7b94c93ab749ec16a04b0dc in this term.
I20240501 19:56:27.253131 11381 raft_consensus.cc:3037] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.253144 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb00c1acda4d98a90b12d877412045" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:27.253543 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb00c1acda4d98a90b12d877412045" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:27.253687 11644 raft_consensus.cc:2379] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ee60ead7b94c93ab749ec16a04b0dc in current term 1: Already voted for candidate 8ad18cd54c6340fd92b874157875fa49 in this term.
I20240501 19:56:27.253921 11182 leader_election.cc:302] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc; no voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:27.254137 12585 raft_consensus.cc:2726] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 19:56:27.254155 11381 raft_consensus.cc:2454] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:27.254392 11579 leader_election.cc:302] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters: 10ee60ead7b94c93ab749ec16a04b0dc
I20240501 19:56:27.254544 12920 raft_consensus.cc:2781] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:27.254624 12920 raft_consensus.cc:686] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:27.254760 12920 consensus_queue.cc:237] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:27.256599 11105 catalog_manager.cc:5484] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:27.326017 12920 raft_consensus.cc:482] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:27.326157 12920 raft_consensus.cc:504] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.326706 12920 leader_election.cc:288] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:27.326884 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cd411ab06242d7809d56cd138fd7a5" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:27.327015 11249 raft_consensus.cc:2454] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:27.327006 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cd411ab06242d7809d56cd138fd7a5" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:27.327132 11512 raft_consensus.cc:2454] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:27.327297 11577 leader_election.cc:302] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:27.327490 12920 raft_consensus.cc:2781] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:27.327548 12920 raft_consensus.cc:482] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:27.327591 12920 raft_consensus.cc:3037] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.328374 12920 raft_consensus.cc:504] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.328662 12920 leader_election.cc:288] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:27.328827 12585 raft_consensus.cc:482] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:27.328966 12585 raft_consensus.cc:504] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:27.328984 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cd411ab06242d7809d56cd138fd7a5" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:27.329018 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cd411ab06242d7809d56cd138fd7a5" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:27.329107 11249 raft_consensus.cc:3037] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.329133 11512 raft_consensus.cc:3037] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.329597 12585 leader_election.cc:288] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:27.329679 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6070b62336054ef4b0444f27844b9ee1" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:27.329789 11381 raft_consensus.cc:2454] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:27.330035 11249 raft_consensus.cc:2454] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:27.330060 11512 raft_consensus.cc:2454] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:27.330097 11183 leader_election.cc:302] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:27.330250 12585 raft_consensus.cc:2781] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:27.330296 12585 raft_consensus.cc:482] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:27.330345 12585 raft_consensus.cc:3037] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.330317 11577 leader_election.cc:302] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:27.330467 12920 raft_consensus.cc:2781] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:27.330561 12920 raft_consensus.cc:686] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:27.330696 12920 consensus_queue.cc:237] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.330737 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6070b62336054ef4b0444f27844b9ee1" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:27.330863 11644 raft_consensus.cc:2454] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:27.331835 12585 raft_consensus.cc:504] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:27.332405 12585 leader_election.cc:288] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:27.332456 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6070b62336054ef4b0444f27844b9ee1" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:27.332468 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6070b62336054ef4b0444f27844b9ee1" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:27.332538 11644 raft_consensus.cc:3037] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.332587 11381 raft_consensus.cc:3037] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.332800 11105 catalog_manager.cc:5484] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:27.333150 11644 raft_consensus.cc:2454] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:27.333453 11182 leader_election.cc:302] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:27.333518 11381 raft_consensus.cc:2454] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:27.333619 12585 raft_consensus.cc:2781] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:27.333699 12585 raft_consensus.cc:686] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:27.333807 12585 consensus_queue.cc:237] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:27.336206 11105 catalog_manager.cc:5484] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:27.500193 12938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:27.500448 12938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:27.503607 12938 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 19:56:27.581929 11381 raft_consensus.cc:1261] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.582290 11249 raft_consensus.cc:1261] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.582829 12920 consensus_queue.cc:1035] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.583042 11249 raft_consensus.cc:1261] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.583137 12920 consensus_queue.cc:1035] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.583250 11512 raft_consensus.cc:1261] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.583757 12920 consensus_queue.cc:1035] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.583901 12784 consensus_queue.cc:1035] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.588256 11378 raft_consensus.cc:1261] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.588718 12585 consensus_queue.cc:1035] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.588836 11644 raft_consensus.cc:1261] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:27.589151 12585 consensus_queue.cc:1035] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:27.614763 12981 mvcc.cc:205] Tried to move back new op lower bound from 7022974515558313984 to 7022974514523320320. Current Snapshot: MvccSnapshot[applied={T|T < 7022974515558313984}]
INSERT report
rows total: 2000
time total: 107.926 ms
time per row: 0.0539628 ms
I20240501 19:56:27.653312 11105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:27.659104 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 0419fba851684c9b9cf0e6096de78a03 (DEFAULT_TABLE table=rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.659420 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 0419fba851684c9b9cf0e6096de78a03 (DEFAULT_TABLE table=rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.659734 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0419fba851684c9b9cf0e6096de78a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.659736 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 1772073172764f8db95c597f8da14f12 (DEFAULT_TABLE table=rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.660014 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1772073172764f8db95c597f8da14f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.660166 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 6326c660e6cb4eb289634a68fc50c4af (DEFAULT_TABLE table=rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.660230 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 1772073172764f8db95c597f8da14f12 (DEFAULT_TABLE table=rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.660491 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1772073172764f8db95c597f8da14f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.660931 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 6326c660e6cb4eb289634a68fc50c4af (DEFAULT_TABLE table=rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.661161 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6326c660e6cb4eb289634a68fc50c4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.662283 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0419fba851684c9b9cf0e6096de78a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.662624 12999 tablet_bootstrap.cc:492] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:27.663245 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 0419fba851684c9b9cf0e6096de78a03 (DEFAULT_TABLE table=rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.663513 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0419fba851684c9b9cf0e6096de78a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.663633 12999 tablet_bootstrap.cc:654] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.663658 13000 tablet_bootstrap.cc:492] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:27.664515 13000 tablet_bootstrap.cc:654] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.664708 13003 tablet_bootstrap.cc:492] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:27.665129 13001 tablet_bootstrap.cc:492] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:27.665575 13003 tablet_bootstrap.cc:654] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.665904 13001 tablet_bootstrap.cc:654] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.665899 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 1772073172764f8db95c597f8da14f12 (DEFAULT_TABLE table=rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.666046 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6326c660e6cb4eb289634a68fc50c4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.666141 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1772073172764f8db95c597f8da14f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.666935 13001 tablet_bootstrap.cc:492] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:27.666936 13000 tablet_bootstrap.cc:492] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:27.667021 13000 ts_tablet_manager.cc:1367] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:27.667057 13001 ts_tablet_manager.cc:1367] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:27.667502 13000 raft_consensus.cc:348] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.667508 13001 raft_consensus.cc:348] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.667609 13000 raft_consensus.cc:374] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.667609 13001 raft_consensus.cc:374] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.667640 13001 raft_consensus.cc:729] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.667640 13000 raft_consensus.cc:729] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.667790 13000 consensus_queue.cc:260] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.667790 13001 consensus_queue.cc:260] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.665899 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 6326c660e6cb4eb289634a68fc50c4af (DEFAULT_TABLE table=rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:27.668151 13000 ts_tablet_manager.cc:1398] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.668279 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6326c660e6cb4eb289634a68fc50c4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.668295 13000 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:27.669476 13000 tablet_bootstrap.cc:654] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.670311 12999 tablet_bootstrap.cc:492] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:27.670518 12999 ts_tablet_manager.cc:1367] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 19:56:27.671098 13001 ts_tablet_manager.cc:1398] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:27.671118 12999 raft_consensus.cc:348] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.671371 12999 raft_consensus.cc:374] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.671487 12999 raft_consensus.cc:729] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.671672 12999 consensus_queue.cc:260] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.671892 13001 tablet_bootstrap.cc:492] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:27.672545 12999 ts_tablet_manager.cc:1398] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:27.672758 12999 tablet_bootstrap.cc:492] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:27.672811 13001 tablet_bootstrap.cc:654] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.673609 12999 tablet_bootstrap.cc:654] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.672369 13000 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:27.673734 13000 ts_tablet_manager.cc:1367] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:27.673992 13001 tablet_bootstrap.cc:492] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:27.674079 13001 ts_tablet_manager.cc:1367] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:27.674130 13000 raft_consensus.cc:348] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.674232 13000 raft_consensus.cc:374] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.674263 13000 raft_consensus.cc:729] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.674378 13000 consensus_queue.cc:260] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.674716 13001 raft_consensus.cc:348] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.674821 13001 raft_consensus.cc:374] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.674854 13001 raft_consensus.cc:729] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.674998 13001 consensus_queue.cc:260] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.675153 13000 ts_tablet_manager.cc:1398] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.675261 13001 ts_tablet_manager.cc:1398] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.675664 12999 tablet_bootstrap.cc:492] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:27.675748 12999 ts_tablet_manager.cc:1367] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:27.676180 12999 raft_consensus.cc:348] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.676285 12999 raft_consensus.cc:374] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.676312 12999 raft_consensus.cc:729] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.676434 12999 consensus_queue.cc:260] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.676700 12999 ts_tablet_manager.cc:1398] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:27.676859 12999 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:27.677654 12999 tablet_bootstrap.cc:654] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.678611 12999 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:27.678694 12999 ts_tablet_manager.cc:1367] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:27.679061 13003 tablet_bootstrap.cc:492] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:27.679153 13003 ts_tablet_manager.cc:1367] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:56:27.679100 12999 raft_consensus.cc:348] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.679198 12999 raft_consensus.cc:374] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.679236 12999 raft_consensus.cc:729] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.679365 12999 consensus_queue.cc:260] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.679744 12999 ts_tablet_manager.cc:1398] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.680089 13003 raft_consensus.cc:348] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.680184 13003 raft_consensus.cc:374] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.680212 13003 raft_consensus.cc:729] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.680331 13003 consensus_queue.cc:260] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.680708 13003 ts_tablet_manager.cc:1398] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:27.680912 13003 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:27.682123 13003 tablet_bootstrap.cc:654] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:27.683214 13003 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:27.683306 13003 ts_tablet_manager.cc:1367] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:27.683753 13003 raft_consensus.cc:348] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.683858 13003 raft_consensus.cc:374] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:27.683918 13003 raft_consensus.cc:729] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:27.684078 13003 consensus_queue.cc:260] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.684353 13003 ts_tablet_manager.cc:1398] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:27.783782 12784 raft_consensus.cc:482] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:27.783901 12784 raft_consensus.cc:504] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.784267 12784 leader_election.cc:288] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:27.784615 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1772073172764f8db95c597f8da14f12" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:27.784682 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1772073172764f8db95c597f8da14f12" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:27.784755 11512 raft_consensus.cc:2454] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:27.784827 11248 raft_consensus.cc:2454] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:27.785051 11577 leader_election.cc:302] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:27.785202 12784 raft_consensus.cc:2781] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:27.785261 12784 raft_consensus.cc:482] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:27.785287 12784 raft_consensus.cc:3037] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.786090 12784 raft_consensus.cc:504] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.786388 12784 leader_election.cc:288] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:27.786723 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1772073172764f8db95c597f8da14f12" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:27.786772 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1772073172764f8db95c597f8da14f12" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:27.786832 11512 raft_consensus.cc:3037] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.786877 11248 raft_consensus.cc:3037] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.787616 11512 raft_consensus.cc:2454] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:27.787678 11248 raft_consensus.cc:2454] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:27.787895 11577 leader_election.cc:302] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:27.788070 12784 raft_consensus.cc:2781] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:27.788159 12784 raft_consensus.cc:686] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:27.788295 12784 consensus_queue.cc:237] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:27.790182 11102 catalog_manager.cc:5484] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:27.834190 12694 raft_consensus.cc:482] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:27.834336 12694 raft_consensus.cc:504] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.834738 12694 leader_election.cc:288] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:27.835093 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6326c660e6cb4eb289634a68fc50c4af" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:27.835091 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6326c660e6cb4eb289634a68fc50c4af" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:27.835233 11512 raft_consensus.cc:2454] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:27.835233 11643 raft_consensus.cc:2454] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:27.835527 11314 leader_election.cc:302] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:27.835706 12694 raft_consensus.cc:2781] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:27.835778 12694 raft_consensus.cc:482] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:27.835805 12694 raft_consensus.cc:3037] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.836637 12694 raft_consensus.cc:504] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.836963 12694 leader_election.cc:288] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:27.837282 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6326c660e6cb4eb289634a68fc50c4af" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:27.837282 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6326c660e6cb4eb289634a68fc50c4af" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:27.837409 11643 raft_consensus.cc:3037] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.837409 11512 raft_consensus.cc:3037] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:27.838352 11643 raft_consensus.cc:2454] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:27.838352 11512 raft_consensus.cc:2454] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:27.838641 11313 leader_election.cc:302] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:27.838814 12694 raft_consensus.cc:2781] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:27.838917 12694 raft_consensus.cc:686] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:27.839061 12694 consensus_queue.cc:237] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:27.840891 11102 catalog_manager.cc:5484] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:28.058594 12934 raft_consensus.cc:482] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:28.058732 12934 raft_consensus.cc:504] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:28.059159 12934 leader_election.cc:288] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:28.059590 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0419fba851684c9b9cf0e6096de78a03" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:28.059604 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0419fba851684c9b9cf0e6096de78a03" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:28.059724 11512 raft_consensus.cc:2454] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:28.059733 11381 raft_consensus.cc:2454] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:28.060014 11181 leader_election.cc:302] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:28.060175 12934 raft_consensus.cc:2781] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:28.060233 12934 raft_consensus.cc:482] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:28.060277 12934 raft_consensus.cc:3037] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.063769 12934 raft_consensus.cc:504] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:28.064149 12934 leader_election.cc:288] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:28.064527 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0419fba851684c9b9cf0e6096de78a03" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:28.064525 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0419fba851684c9b9cf0e6096de78a03" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:28.064640 11512 raft_consensus.cc:3037] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.064651 11381 raft_consensus.cc:3037] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.065563 11381 raft_consensus.cc:2454] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:28.065568 11512 raft_consensus.cc:2454] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:28.065899 11183 leader_election.cc:302] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:28.066139 12992 raft_consensus.cc:2781] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:28.066239 12992 raft_consensus.cc:686] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:28.066412 12992 consensus_queue.cc:237] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:28.068291 11102 catalog_manager.cc:5484] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:28.236675 13016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:28.236945 13016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:28.240074 13016 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 19:56:28.309278 12694 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:28.310252 12784 consensus_queue.cc:1035] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:28.314061 12835 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:28.320320 13045 consensus_queue.cc:1035] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:28.338398 11512 raft_consensus.cc:1261] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:28.342988 12585 consensus_queue.cc:1035] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:28.345248 11381 raft_consensus.cc:1261] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:28.345690 12585 consensus_queue.cc:1035] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.454 ms
time per row: 0.0602269 ms
I20240501 19:56:28.403599 11106 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:28.409297 11358 tablet_service.cc:1467] Processing CreateTablet for tablet b40fd5ed372743a1bb0680bd7de634fb (DEFAULT_TABLE table=rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:28.409638 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40fd5ed372743a1bb0680bd7de634fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:28.409777 11492 tablet_service.cc:1467] Processing CreateTablet for tablet b40fd5ed372743a1bb0680bd7de634fb (DEFAULT_TABLE table=rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:28.410087 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40fd5ed372743a1bb0680bd7de634fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:28.410326 11623 tablet_service.cc:1467] Processing CreateTablet for tablet d9e0fc705bd4427ab966ff1edfae278b (DEFAULT_TABLE table=rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:28.410578 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e0fc705bd4427ab966ff1edfae278b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:28.411600 13080 tablet_bootstrap.cc:492] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:28.411828 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 1b74fc0672f34efaaee13512df8577ac (DEFAULT_TABLE table=rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:28.412045 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b74fc0672f34efaaee13512df8577ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:28.412634 13080 tablet_bootstrap.cc:654] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:28.412845 11227 tablet_service.cc:1467] Processing CreateTablet for tablet b40fd5ed372743a1bb0680bd7de634fb (DEFAULT_TABLE table=rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:28.413090 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40fd5ed372743a1bb0680bd7de634fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:28.414124 11358 tablet_service.cc:1467] Processing CreateTablet for tablet d9e0fc705bd4427ab966ff1edfae278b (DEFAULT_TABLE table=rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:28.414327 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e0fc705bd4427ab966ff1edfae278b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:28.415073 13080 tablet_bootstrap.cc:492] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:28.415154 13080 ts_tablet_manager.cc:1367] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:28.415369 11228 tablet_service.cc:1467] Processing CreateTablet for tablet d9e0fc705bd4427ab966ff1edfae278b (DEFAULT_TABLE table=rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:28.415728 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e0fc705bd4427ab966ff1edfae278b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:28.415724 13080 raft_consensus.cc:348] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:28.415863 13080 raft_consensus.cc:374] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:28.415896 13080 raft_consensus.cc:729] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:28.416043 13080 consensus_queue.cc:260] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:28.416464 13081 tablet_bootstrap.cc:492] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:28.417083 13080 ts_tablet_manager.cc:1398] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:28.417248 13080 tablet_bootstrap.cc:492] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:28.417541 13081 tablet_bootstrap.cc:654] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:28.417826 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 1b74fc0672f34efaaee13512df8577ac (DEFAULT_TABLE table=rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:28.418123 13080 tablet_bootstrap.cc:654] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:28.418119 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b74fc0672f34efaaee13512df8577ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:28.418663 13081 tablet_bootstrap.cc:492] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:28.418751 13081 ts_tablet_manager.cc:1367] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:28.419138 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 1b74fc0672f34efaaee13512df8577ac (DEFAULT_TABLE table=rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:28.419178 13081 raft_consensus.cc:348] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:28.419275 13081 raft_consensus.cc:374] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:28.419306 13081 raft_consensus.cc:729] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:28.419360 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b74fc0672f34efaaee13512df8577ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:28.419418 13081 consensus_queue.cc:260] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:28.419728 13081 ts_tablet_manager.cc:1398] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:28.420432 13081 tablet_bootstrap.cc:492] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:28.420634 13084 tablet_bootstrap.cc:492] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:28.420666 13080 tablet_bootstrap.cc:492] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:28.420742 13080 ts_tablet_manager.cc:1367] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:28.421142 13080 raft_consensus.cc:348] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:28.421233 13080 raft_consensus.cc:374] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:28.421252 13081 tablet_bootstrap.cc:654] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:28.421281 13080 raft_consensus.cc:729] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:28.421412 13084 tablet_bootstrap.cc:654] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:28.421420 13080 consensus_queue.cc:260] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:28.421728 13080 ts_tablet_manager.cc:1398] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:28.422567 13081 tablet_bootstrap.cc:492] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:28.422559 13082 tablet_bootstrap.cc:492] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:28.422642 13081 ts_tablet_manager.cc:1367] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:28.423194 13081 raft_consensus.cc:348] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:28.423312 13081 raft_consensus.cc:374] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:28.423343 13081 raft_consensus.cc:729] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:28.423473 13082 tablet_bootstrap.cc:654] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:28.423563 13081 consensus_queue.cc:260] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:28.423854 13081 ts_tablet_manager.cc:1398] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:28.425096 13084 tablet_bootstrap.cc:492] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:28.425232 13084 ts_tablet_manager.cc:1367] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:28.425555 13082 tablet_bootstrap.cc:492] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:28.425693 13082 ts_tablet_manager.cc:1367] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:28.425802 13084 raft_consensus.cc:348] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:28.425917 13084 raft_consensus.cc:374] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:28.425956 13084 raft_consensus.cc:729] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:28.426077 13084 consensus_queue.cc:260] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:28.426424 13084 ts_tablet_manager.cc:1398] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:28.426649 13084 tablet_bootstrap.cc:492] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:28.426828 13082 raft_consensus.cc:348] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:28.426944 13082 raft_consensus.cc:374] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:28.427006 13082 raft_consensus.cc:729] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:28.427136 13082 consensus_queue.cc:260] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:28.427390 13082 ts_tablet_manager.cc:1398] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:28.427758 13084 tablet_bootstrap.cc:654] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:28.428781 13082 tablet_bootstrap.cc:492] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:28.428932 13084 tablet_bootstrap.cc:492] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:28.429021 13084 ts_tablet_manager.cc:1367] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:28.429548 13084 raft_consensus.cc:348] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:28.429636 13084 raft_consensus.cc:374] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:28.429656 13084 raft_consensus.cc:729] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:28.429765 13084 consensus_queue.cc:260] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:28.430080 13084 ts_tablet_manager.cc:1398] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:28.430238 13084 tablet_bootstrap.cc:492] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:28.430778 13082 tablet_bootstrap.cc:654] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:28.431032 13084 tablet_bootstrap.cc:654] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:28.432058 13082 tablet_bootstrap.cc:492] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:28.432132 13082 ts_tablet_manager.cc:1367] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:28.432274 13084 tablet_bootstrap.cc:492] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:28.432359 13084 ts_tablet_manager.cc:1367] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:28.432610 13082 raft_consensus.cc:348] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:28.432737 13082 raft_consensus.cc:374] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:28.432710 13084 raft_consensus.cc:348] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:28.432782 13082 raft_consensus.cc:729] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:28.432803 13084 raft_consensus.cc:374] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:28.432842 13084 raft_consensus.cc:729] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:28.432922 13082 consensus_queue.cc:260] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:28.432943 13084 consensus_queue.cc:260] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:28.433209 13082 ts_tablet_manager.cc:1398] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:28.433211 13084 ts_tablet_manager.cc:1398] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:28.521378 13014 raft_consensus.cc:482] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:28.521521 13014 raft_consensus.cc:504] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:28.521917 13014 leader_election.cc:288] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:28.522244 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40fd5ed372743a1bb0680bd7de634fb" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:28.522315 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40fd5ed372743a1bb0680bd7de634fb" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:28.522420 11381 raft_consensus.cc:2454] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:28.522450 11248 raft_consensus.cc:2454] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:28.522737 11447 leader_election.cc:302] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:28.522919 13014 raft_consensus.cc:2781] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:28.522980 13014 raft_consensus.cc:482] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:28.523023 13014 raft_consensus.cc:3037] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.523933 13014 raft_consensus.cc:504] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:28.524237 13014 leader_election.cc:288] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:28.524549 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40fd5ed372743a1bb0680bd7de634fb" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:28.524549 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40fd5ed372743a1bb0680bd7de634fb" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:28.524669 11248 raft_consensus.cc:3037] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.524665 11381 raft_consensus.cc:3037] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.525409 11381 raft_consensus.cc:2454] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:28.525601 11248 raft_consensus.cc:2454] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:28.525681 11447 leader_election.cc:302] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:28.525846 13014 raft_consensus.cc:2781] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:28.525941 13014 raft_consensus.cc:686] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:28.526074 13014 consensus_queue.cc:237] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:28.527956 11106 catalog_manager.cc:5484] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:28.651568 13048 raft_consensus.cc:482] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:28.651712 13048 raft_consensus.cc:504] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:28.652165 13048 leader_election.cc:288] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:28.652645 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b74fc0672f34efaaee13512df8577ac" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:28.652662 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b74fc0672f34efaaee13512df8577ac" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:28.652802 11512 raft_consensus.cc:2454] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:28.652824 11248 raft_consensus.cc:2454] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:28.653112 11577 leader_election.cc:302] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:28.653292 13048 raft_consensus.cc:2781] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:28.653349 13048 raft_consensus.cc:482] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:28.653378 13048 raft_consensus.cc:3037] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.654273 13048 raft_consensus.cc:504] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:28.654572 13048 leader_election.cc:288] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:28.654882 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b74fc0672f34efaaee13512df8577ac" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:28.654924 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b74fc0672f34efaaee13512df8577ac" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:28.655001 11512 raft_consensus.cc:3037] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.655019 11248 raft_consensus.cc:3037] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.655939 11512 raft_consensus.cc:2454] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:28.655939 11248 raft_consensus.cc:2454] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:28.656246 11577 leader_election.cc:302] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:28.656399 13048 raft_consensus.cc:2781] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:28.656489 13048 raft_consensus.cc:686] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:28.656623 13048 consensus_queue.cc:237] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:28.658377 11106 catalog_manager.cc:5484] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:28.730952 12585 raft_consensus.cc:482] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:28.731076 12585 raft_consensus.cc:504] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:28.731465 12585 leader_election.cc:288] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:28.731758 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e0fc705bd4427ab966ff1edfae278b" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:28.731770 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e0fc705bd4427ab966ff1edfae278b" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:28.731879 11643 raft_consensus.cc:2454] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:28.731910 11381 raft_consensus.cc:2454] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:28.732159 11182 leader_election.cc:302] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:28.732313 12585 raft_consensus.cc:2781] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:28.732383 12585 raft_consensus.cc:482] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:28.732417 12585 raft_consensus.cc:3037] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.733165 12585 raft_consensus.cc:504] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:28.733491 12585 leader_election.cc:288] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:28.733794 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e0fc705bd4427ab966ff1edfae278b" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:28.733793 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e0fc705bd4427ab966ff1edfae278b" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:28.733897 11643 raft_consensus.cc:3037] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.733892 11381 raft_consensus.cc:3037] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:28.734649 11643 raft_consensus.cc:2454] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:28.734653 11381 raft_consensus.cc:2454] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:28.734915 11182 leader_election.cc:302] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:28.735064 12585 raft_consensus.cc:2781] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:28.735150 12585 raft_consensus.cc:686] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:28.735287 12585 consensus_queue.cc:237] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:28.737100 11106 catalog_manager.cc:5484] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:28.888919 13096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:28.889184 13096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:28.892288 13096 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240501 19:56:28.979892 11643 raft_consensus.cc:1261] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:28.979866 11381 raft_consensus.cc:1261] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:28.980311 12585 consensus_queue.cc:1035] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:28.980641 12585 consensus_queue.cc:1035] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:28.984398 13133 mvcc.cc:205] Tried to move back new op lower bound from 7022974521256734720 to 7022974520261148672. Current Snapshot: MvccSnapshot[applied={T|T < 7022974521256734720}]
I20240501 19:56:28.989768 11248 raft_consensus.cc:1261] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:28.990160 13014 consensus_queue.cc:1035] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:28.991991 11381 raft_consensus.cc:1261] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:28.992344 13014 consensus_queue.cc:1035] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:28.995044 13131 mvcc.cc:205] Tried to move back new op lower bound from 7022974521315717120 to 7022974520261148672. Current Snapshot: MvccSnapshot[applied={T|T < 7022974521256734720}]
I20240501 19:56:28.999115 11249 raft_consensus.cc:1261] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:28.999514 13048 consensus_queue.cc:1035] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:29.004704 11512 raft_consensus.cc:1261] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:29.005211 13048 consensus_queue.cc:1035] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:29.011130 13126 mvcc.cc:205] Tried to move back new op lower bound from 7022974521336516608 to 7022974519938777088. Current Snapshot: MvccSnapshot[applied={T|T < 7022974521336516608}]
INSERT report
rows total: 2000
time total: 117.69 ms
time per row: 0.0588448 ms
I20240501 19:56:29.051208 11106 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:29.056910 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 61cadb3ab334435eaebe99fc3c9e496b (DEFAULT_TABLE table=rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.056990 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 61cadb3ab334435eaebe99fc3c9e496b (DEFAULT_TABLE table=rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.057230 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cadb3ab334435eaebe99fc3c9e496b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.057287 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cadb3ab334435eaebe99fc3c9e496b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.057837 11360 tablet_service.cc:1467] Processing CreateTablet for tablet e307d45fb57f444ca09d94d59b9bcd64 (DEFAULT_TABLE table=rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.058112 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e307d45fb57f444ca09d94d59b9bcd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.059231 13159 tablet_bootstrap.cc:492] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:29.059643 11491 tablet_service.cc:1467] Processing CreateTablet for tablet eff09e5884f44e18a66583e8e0e73679 (DEFAULT_TABLE table=rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.059682 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 61cadb3ab334435eaebe99fc3c9e496b (DEFAULT_TABLE table=rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.059854 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff09e5884f44e18a66583e8e0e73679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.059908 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cadb3ab334435eaebe99fc3c9e496b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.060204 13159 tablet_bootstrap.cc:654] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.057849 11359 tablet_service.cc:1467] Processing CreateTablet for tablet eff09e5884f44e18a66583e8e0e73679 (DEFAULT_TABLE table=rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.060525 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff09e5884f44e18a66583e8e0e73679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.061965 11492 tablet_service.cc:1467] Processing CreateTablet for tablet e307d45fb57f444ca09d94d59b9bcd64 (DEFAULT_TABLE table=rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.062199 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e307d45fb57f444ca09d94d59b9bcd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.062539 11624 tablet_service.cc:1467] Processing CreateTablet for tablet e307d45fb57f444ca09d94d59b9bcd64 (DEFAULT_TABLE table=rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.062781 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e307d45fb57f444ca09d94d59b9bcd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.063046 13161 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:29.063220 13160 tablet_bootstrap.cc:492] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:29.063897 13161 tablet_bootstrap.cc:654] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.064158 13160 tablet_bootstrap.cc:654] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.065261 13160 tablet_bootstrap.cc:492] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:29.065351 13160 ts_tablet_manager.cc:1367] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:29.066108 13160 raft_consensus.cc:348] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:29.066206 13160 raft_consensus.cc:374] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.066227 13160 raft_consensus.cc:729] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.066323 13160 consensus_queue.cc:260] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:29.066767 11228 tablet_service.cc:1467] Processing CreateTablet for tablet eff09e5884f44e18a66583e8e0e73679 (DEFAULT_TABLE table=rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.067051 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff09e5884f44e18a66583e8e0e73679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.067153 13160 ts_tablet_manager.cc:1398] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:29.067312 13160 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:29.068253 13160 tablet_bootstrap.cc:654] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.069811 13166 tablet_bootstrap.cc:492] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:29.070163 13160 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:29.070242 13160 ts_tablet_manager.cc:1367] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:29.070674 13160 raft_consensus.cc:348] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.070761 13166 tablet_bootstrap.cc:654] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.070775 13160 raft_consensus.cc:374] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.070801 13160 raft_consensus.cc:729] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.070927 13160 consensus_queue.cc:260] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.071192 13160 ts_tablet_manager.cc:1398] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:29.071911 13166 tablet_bootstrap.cc:492] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:29.072021 13166 ts_tablet_manager.cc:1367] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:29.072217 13048 raft_consensus.cc:482] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:29.072296 13048 raft_consensus.cc:504] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.072436 13166 raft_consensus.cc:348] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.072537 13166 raft_consensus.cc:374] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.072567 13166 raft_consensus.cc:729] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.072633 13048 leader_election.cc:288] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:29.072714 13166 consensus_queue.cc:260] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.073367 13159 tablet_bootstrap.cc:492] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:29.073463 13159 ts_tablet_manager.cc:1367] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:56:29.073760 13161 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:29.073851 13161 ts_tablet_manager.cc:1367] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:56:29.074016 13159 raft_consensus.cc:348] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:29.074158 13159 raft_consensus.cc:374] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.074198 13159 raft_consensus.cc:729] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.074275 13161 raft_consensus.cc:348] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.074311 13159 consensus_queue.cc:260] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:29.074364 13161 raft_consensus.cc:374] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.074385 13161 raft_consensus.cc:729] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.074498 13161 consensus_queue.cc:260] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.074589 13159 ts_tablet_manager.cc:1398] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:29.074759 13159 tablet_bootstrap.cc:492] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:29.074774 13161 ts_tablet_manager.cc:1398] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:29.074945 13161 tablet_bootstrap.cc:492] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:29.075865 13161 tablet_bootstrap.cc:654] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.076373 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e307d45fb57f444ca09d94d59b9bcd64" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:29.076395 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e307d45fb57f444ca09d94d59b9bcd64" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:29.076491 11381 raft_consensus.cc:2454] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:29.076575 13159 tablet_bootstrap.cc:654] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.076797 11579 leader_election.cc:302] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:29.076961 13048 raft_consensus.cc:2781] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:29.077003 13048 raft_consensus.cc:482] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:29.077071 13166 ts_tablet_manager.cc:1398] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20240501 19:56:29.077020 11577 leader_election.cc:341] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:29.077035 13048 raft_consensus.cc:3037] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.077539 13161 tablet_bootstrap.cc:492] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:29.077616 13161 ts_tablet_manager.cc:1367] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:29.078047 13161 raft_consensus.cc:348] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:29.078152 13161 raft_consensus.cc:374] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.078183 13161 raft_consensus.cc:729] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.078140 13048 raft_consensus.cc:504] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.078224 13159 tablet_bootstrap.cc:492] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:29.078297 13159 ts_tablet_manager.cc:1367] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:29.078322 13161 consensus_queue.cc:260] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:29.078459 13048 leader_election.cc:288] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:29.078581 13161 ts_tablet_manager.cc:1398] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:29.078729 13161 tablet_bootstrap.cc:492] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:29.078723 13159 raft_consensus.cc:348] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.078809 13159 raft_consensus.cc:374] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.078781 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e307d45fb57f444ca09d94d59b9bcd64" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:29.078836 13159 raft_consensus.cc:729] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.078877 11381 raft_consensus.cc:3037] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.078938 13159 consensus_queue.cc:260] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.079314 13159 ts_tablet_manager.cc:1398] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:29.079685 13161 tablet_bootstrap.cc:654] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.079553 13159 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:29.079835 11381 raft_consensus.cc:2454] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:29.080261 11579 leader_election.cc:302] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:29.080521 13048 raft_consensus.cc:2781] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:29.080596 13048 raft_consensus.cc:686] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:29.080710 13161 tablet_bootstrap.cc:492] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:29.080703 13048 consensus_queue.cc:237] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.080778 13161 ts_tablet_manager.cc:1367] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:29.081171 13161 raft_consensus.cc:348] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.081281 13161 raft_consensus.cc:374] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.081313 13161 raft_consensus.cc:729] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.081415 13161 consensus_queue.cc:260] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.081465 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e307d45fb57f444ca09d94d59b9bcd64" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:29.081701 13161 ts_tablet_manager.cc:1398] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 19:56:29.082265 11577 leader_election.cc:341] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:29.083129 11099 catalog_manager.cc:5484] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:29.084156 13159 tablet_bootstrap.cc:654] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.085194 13159 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:29.085261 13159 ts_tablet_manager.cc:1367] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:29.085587 13159 raft_consensus.cc:348] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.085670 13159 raft_consensus.cc:374] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.085690 13159 raft_consensus.cc:729] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.085788 13159 consensus_queue.cc:260] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.086030 13159 ts_tablet_manager.cc:1398] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:29.110738 13155 raft_consensus.cc:482] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:29.110858 13155 raft_consensus.cc:504] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.111258 13155 leader_election.cc:288] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:29.111608 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e307d45fb57f444ca09d94d59b9bcd64" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:29.111617 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e307d45fb57f444ca09d94d59b9bcd64" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:29.111780 11381 raft_consensus.cc:2379] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08711534813d47ba85aaec0771ed5d0f in current term 1: Already voted for candidate 8ad18cd54c6340fd92b874157875fa49 in this term.
I20240501 19:56:29.112037 11447 leader_election.cc:302] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f; no voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:29.112214 13155 raft_consensus.cc:3037] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.113065 13155 raft_consensus.cc:2726] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:29.262954 13158 raft_consensus.cc:482] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:29.263127 13158 raft_consensus.cc:504] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:29.263639 13158 leader_election.cc:288] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:29.263924 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cadb3ab334435eaebe99fc3c9e496b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:29.264081 11512 raft_consensus.cc:2454] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:29.264139 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cadb3ab334435eaebe99fc3c9e496b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:29.264264 11643 raft_consensus.cc:2454] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:29.264350 11313 leader_election.cc:302] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:29.264526 13158 raft_consensus.cc:2781] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:29.264585 13158 raft_consensus.cc:482] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:29.264629 13158 raft_consensus.cc:3037] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.265460 13158 raft_consensus.cc:504] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:29.265748 13158 leader_election.cc:288] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:29.266042 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cadb3ab334435eaebe99fc3c9e496b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:29.266068 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cadb3ab334435eaebe99fc3c9e496b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:29.266141 11512 raft_consensus.cc:3037] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.266161 11643 raft_consensus.cc:3037] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.266839 11512 raft_consensus.cc:2454] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:29.266850 11643 raft_consensus.cc:2454] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:29.267105 11313 leader_election.cc:302] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:29.267261 13158 raft_consensus.cc:2781] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:29.267350 13158 raft_consensus.cc:686] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:29.267499 13158 consensus_queue.cc:237] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:29.269412 11099 catalog_manager.cc:5484] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:29.271374 13154 raft_consensus.cc:482] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:29.271483 13154 raft_consensus.cc:504] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.271844 13154 leader_election.cc:288] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:29.272156 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff09e5884f44e18a66583e8e0e73679" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:29.272286 11249 raft_consensus.cc:2454] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:29.272511 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff09e5884f44e18a66583e8e0e73679" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:29.272539 11445 leader_election.cc:302] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:29.272619 11381 raft_consensus.cc:2454] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:29.272671 13154 raft_consensus.cc:2781] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:29.272711 13154 raft_consensus.cc:482] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:29.272734 13154 raft_consensus.cc:3037] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.273474 13154 raft_consensus.cc:504] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.273772 13154 leader_election.cc:288] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:29.274097 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff09e5884f44e18a66583e8e0e73679" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:29.274118 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff09e5884f44e18a66583e8e0e73679" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:29.274197 11381 raft_consensus.cc:3037] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.274209 11249 raft_consensus.cc:3037] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.274857 11381 raft_consensus.cc:2454] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:29.274924 11249 raft_consensus.cc:2454] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:29.275123 11447 leader_election.cc:302] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:29.275274 13154 raft_consensus.cc:2781] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:29.275355 13154 raft_consensus.cc:686] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:29.275512 13154 consensus_queue.cc:237] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.277246 11099 catalog_manager.cc:5484] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:29.421463 13175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:29.421734 13175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:29.424938 13175 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240501 19:56:29.510401 11512 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:29.510427 11381 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:29.510753 11644 raft_consensus.cc:1261] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:29.511291 11512 raft_consensus.cc:1261] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:29.511370 13048 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:29.511683 13048 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:29.511863 13158 consensus_queue.cc:1035] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:29.515856 11381 raft_consensus.cc:1261] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:29.516228 11248 raft_consensus.cc:1261] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:29.516474 13154 consensus_queue.cc:1035] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:29.516793 13154 consensus_queue.cc:1035] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:29.518424 12835 consensus_queue.cc:1035] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.635 ms
time per row: 0.0638175 ms
I20240501 19:56:29.595901 11103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:29.601720 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1f6382bcc84fcda444dcfae87b00c1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.602093 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1f6382bcc84fcda444dcfae87b00c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.602155 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 85ba57eb6aae4f12b4a4641d633104a0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.602419 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ba57eb6aae4f12b4a4641d633104a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.602567 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 85ba57eb6aae4f12b4a4641d633104a0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.602859 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ba57eb6aae4f12b4a4641d633104a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.604475 13232 tablet_bootstrap.cc:492] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:29.604657 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1f6382bcc84fcda444dcfae87b00c1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.604658 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1f6382bcc84fcda444dcfae87b00c1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.602567 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 582d72e630c84088a6e4a7309ae7e952 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.604887 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1f6382bcc84fcda444dcfae87b00c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.604899 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1f6382bcc84fcda444dcfae87b00c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.605031 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582d72e630c84088a6e4a7309ae7e952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.605305 11622 tablet_service.cc:1467] Processing CreateTablet for tablet 582d72e630c84088a6e4a7309ae7e952 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.605485 13232 tablet_bootstrap.cc:654] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.605515 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582d72e630c84088a6e4a7309ae7e952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.606515 13232 tablet_bootstrap.cc:492] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:29.606600 13232 ts_tablet_manager.cc:1367] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:29.607009 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 85ba57eb6aae4f12b4a4641d633104a0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.607059 13232 raft_consensus.cc:348] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.607167 13232 raft_consensus.cc:374] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.607196 13232 raft_consensus.cc:729] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.607213 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ba57eb6aae4f12b4a4641d633104a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.607314 13232 consensus_queue.cc:260] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.607739 13233 tablet_bootstrap.cc:492] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:29.608232 13232 ts_tablet_manager.cc:1398] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:29.608639 13233 tablet_bootstrap.cc:654] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.609654 13233 tablet_bootstrap.cc:492] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:29.609655 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 582d72e630c84088a6e4a7309ae7e952 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:29.609750 13233 ts_tablet_manager.cc:1367] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:29.609912 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582d72e630c84088a6e4a7309ae7e952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.610206 13233 raft_consensus.cc:348] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.610317 13233 raft_consensus.cc:374] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.610347 13233 raft_consensus.cc:729] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.610460 13233 consensus_queue.cc:260] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.610735 13233 ts_tablet_manager.cc:1398] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:29.612169 13234 tablet_bootstrap.cc:492] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:29.612591 13232 tablet_bootstrap.cc:492] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:29.612949 13234 tablet_bootstrap.cc:654] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.613415 13232 tablet_bootstrap.cc:654] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.613498 13236 tablet_bootstrap.cc:492] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:29.614380 13236 tablet_bootstrap.cc:654] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.614511 13232 tablet_bootstrap.cc:492] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:29.614602 13232 ts_tablet_manager.cc:1367] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:29.615219 13234 tablet_bootstrap.cc:492] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:29.615208 13232 raft_consensus.cc:348] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.615305 13234 ts_tablet_manager.cc:1367] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:29.615324 13232 raft_consensus.cc:374] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.615371 13232 raft_consensus.cc:729] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.615530 13232 consensus_queue.cc:260] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.615796 13234 raft_consensus.cc:348] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.615914 13232 ts_tablet_manager.cc:1398] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:29.615953 13234 raft_consensus.cc:374] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.616065 13234 raft_consensus.cc:729] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.616243 13234 consensus_queue.cc:260] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.616539 13234 ts_tablet_manager.cc:1398] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:29.617357 13236 tablet_bootstrap.cc:492] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:29.617574 13234 tablet_bootstrap.cc:492] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:29.618374 13234 tablet_bootstrap.cc:654] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.618814 13236 ts_tablet_manager.cc:1367] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:29.619230 13236 raft_consensus.cc:348] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.619338 13236 raft_consensus.cc:374] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.619398 13236 raft_consensus.cc:729] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.619666 13236 consensus_queue.cc:260] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.620054 13236 ts_tablet_manager.cc:1398] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:29.620520 13234 tablet_bootstrap.cc:492] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:29.620635 13234 ts_tablet_manager.cc:1367] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:29.621003 13234 raft_consensus.cc:348] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.621090 13234 raft_consensus.cc:374] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.621119 13234 raft_consensus.cc:729] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.621235 13234 consensus_queue.cc:260] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.621608 13234 ts_tablet_manager.cc:1398] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:29.622166 13234 tablet_bootstrap.cc:492] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:29.622936 13234 tablet_bootstrap.cc:654] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.623173 13236 tablet_bootstrap.cc:492] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:29.623980 13234 tablet_bootstrap.cc:492] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:29.624064 13234 ts_tablet_manager.cc:1367] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:29.624111 13236 tablet_bootstrap.cc:654] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.624511 13234 raft_consensus.cc:348] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.624629 13234 raft_consensus.cc:374] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.624662 13234 raft_consensus.cc:729] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.624789 13234 consensus_queue.cc:260] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.625070 13234 ts_tablet_manager.cc:1398] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:29.625744 13236 tablet_bootstrap.cc:492] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:29.625826 13236 ts_tablet_manager.cc:1367] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:29.626247 13236 raft_consensus.cc:348] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.626358 13236 raft_consensus.cc:374] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.626396 13236 raft_consensus.cc:729] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.626526 13236 consensus_queue.cc:260] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.626808 13236 ts_tablet_manager.cc:1398] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:29.627007 13236 tablet_bootstrap.cc:492] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:29.628381 13236 tablet_bootstrap.cc:654] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:29.629410 13236 tablet_bootstrap.cc:492] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:29.629488 13236 ts_tablet_manager.cc:1367] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:29.629839 13236 raft_consensus.cc:348] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.629927 13236 raft_consensus.cc:374] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:29.629948 13236 raft_consensus.cc:729] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:29.630092 13236 consensus_queue.cc:260] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.630347 13236 ts_tablet_manager.cc:1398] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:29.643258 13219 raft_consensus.cc:482] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:29.643415 13219 raft_consensus.cc:504] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.643919 13219 leader_election.cc:288] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:29.644248 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582d72e630c84088a6e4a7309ae7e952" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:29.644315 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582d72e630c84088a6e4a7309ae7e952" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:29.644395 11248 raft_consensus.cc:2454] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:29.644479 11378 raft_consensus.cc:2454] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:29.644702 11577 leader_election.cc:302] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:29.644865 13219 raft_consensus.cc:2781] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:29.644924 13219 raft_consensus.cc:482] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:29.644968 13219 raft_consensus.cc:3037] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.645833 13219 raft_consensus.cc:504] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.646168 13219 leader_election.cc:288] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:29.646487 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582d72e630c84088a6e4a7309ae7e952" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:29.646498 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582d72e630c84088a6e4a7309ae7e952" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:29.646607 11378 raft_consensus.cc:3037] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.646612 11248 raft_consensus.cc:3037] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.647374 11248 raft_consensus.cc:2454] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:29.647466 11378 raft_consensus.cc:2454] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:29.647682 11577 leader_election.cc:302] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:29.647833 13219 raft_consensus.cc:2781] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:29.647919 13219 raft_consensus.cc:686] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:29.648064 13219 consensus_queue.cc:237] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.649933 11102 catalog_manager.cc:5484] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:29.688609 13220 raft_consensus.cc:482] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:29.688614 13209 raft_consensus.cc:482] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:29.688746 13209 raft_consensus.cc:504] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.688743 13220 raft_consensus.cc:504] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.689142 13220 leader_election.cc:288] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:29.689167 13209 leader_election.cc:288] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:29.689554 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1f6382bcc84fcda444dcfae87b00c1" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:29.689543 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1f6382bcc84fcda444dcfae87b00c1" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:29.689596 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1f6382bcc84fcda444dcfae87b00c1" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:29.689543 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1f6382bcc84fcda444dcfae87b00c1" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:29.689671 11378 raft_consensus.cc:2454] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:29.689692 11248 raft_consensus.cc:2454] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:29.689708 11643 raft_consensus.cc:2454] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:29.689962 11314 leader_election.cc:302] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:29.690026 11183 leader_election.cc:302] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters: 8ad18cd54c6340fd92b874157875fa49
I20240501 19:56:29.690151 13209 raft_consensus.cc:2781] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:29.690171 13220 raft_consensus.cc:2781] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:29.690205 13209 raft_consensus.cc:482] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:29.690212 13220 raft_consensus.cc:482] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:29.690238 13209 raft_consensus.cc:3037] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.690248 13220 raft_consensus.cc:3037] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.691195 13209 raft_consensus.cc:504] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.691197 13220 raft_consensus.cc:504] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.691556 13209 leader_election.cc:288] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:29.691560 13220 leader_election.cc:288] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:29.691794 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1f6382bcc84fcda444dcfae87b00c1" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:29.691826 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1f6382bcc84fcda444dcfae87b00c1" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:29.691918 11378 raft_consensus.cc:2379] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ee60ead7b94c93ab749ec16a04b0dc in current term 1: Already voted for candidate ef60ab93b49c4346af4f62d3b98558b8 in this term.
I20240501 19:56:29.691943 11248 raft_consensus.cc:2379] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ef60ab93b49c4346af4f62d3b98558b8 in current term 1: Already voted for candidate 10ee60ead7b94c93ab749ec16a04b0dc in this term.
I20240501 19:56:29.691924 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1f6382bcc84fcda444dcfae87b00c1" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:29.692028 11643 raft_consensus.cc:3037] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.692127 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1f6382bcc84fcda444dcfae87b00c1" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:29.692499 11183 leader_election.cc:302] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc; no voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:29.692682 13220 raft_consensus.cc:2726] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 19:56:29.692981 11643 raft_consensus.cc:2454] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:29.693228 11314 leader_election.cc:302] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters: 10ee60ead7b94c93ab749ec16a04b0dc
I20240501 19:56:29.693369 13209 raft_consensus.cc:2781] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:29.693447 13209 raft_consensus.cc:686] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:29.693575 13209 consensus_queue.cc:237] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:29.695366 11102 catalog_manager.cc:5484] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:29.874873 13154 raft_consensus.cc:482] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:29.875017 13154 raft_consensus.cc:504] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.875528 13154 leader_election.cc:288] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:29.875912 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ba57eb6aae4f12b4a4641d633104a0" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:29.875924 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ba57eb6aae4f12b4a4641d633104a0" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:29.876045 11643 raft_consensus.cc:2454] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:29.876051 11248 raft_consensus.cc:2454] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:29.876331 11445 leader_election.cc:302] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:29.876484 13154 raft_consensus.cc:2781] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:29.876541 13154 raft_consensus.cc:482] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:29.876579 13154 raft_consensus.cc:3037] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.877367 13154 raft_consensus.cc:504] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.877683 13154 leader_election.cc:288] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:29.877969 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ba57eb6aae4f12b4a4641d633104a0" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:29.877985 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ba57eb6aae4f12b4a4641d633104a0" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:29.878072 11248 raft_consensus.cc:3037] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.878088 11643 raft_consensus.cc:3037] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:29.878810 11248 raft_consensus.cc:2454] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:29.878823 11643 raft_consensus.cc:2454] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:29.879117 11445 leader_election.cc:302] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:29.879262 13154 raft_consensus.cc:2781] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:29.879348 13154 raft_consensus.cc:686] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:29.879520 13154 consensus_queue.cc:237] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:29.881368 11102 catalog_manager.cc:5484] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:30.017983 13248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:30.018275 13248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:30.021710 13248 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:30.041190 13219 consensus_queue.cc:1035] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:30.044726 13150 consensus_queue.cc:1035] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
I20240501 19:56:30.118952 11249 raft_consensus.cc:1261] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.119374 13209 consensus_queue.cc:1035] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:30.122850 11644 raft_consensus.cc:1261] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.124866 12835 consensus_queue.cc:1035] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:30.127897 11643 raft_consensus.cc:1261] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.128295 13155 consensus_queue.cc:1035] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:30.129792 11248 raft_consensus.cc:1261] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.130209 13155 consensus_queue.cc:1035] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.944 ms
time per row: 0.0579721 ms
I20240501 19:56:30.187556 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:30.193451 11492 tablet_service.cc:1467] Processing CreateTablet for tablet 6f594c53151b4d00a1784b6366e2814a (DEFAULT_TABLE table=rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.193540 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 6f594c53151b4d00a1784b6366e2814a (DEFAULT_TABLE table=rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.193780 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f594c53151b4d00a1784b6366e2814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.193781 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f594c53151b4d00a1784b6366e2814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.194223 11358 tablet_service.cc:1467] Processing CreateTablet for tablet e6c1d3814e364e718449c3e4379b31ed (DEFAULT_TABLE table=rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.194434 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c1d3814e364e718449c3e4379b31ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.195765 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 6f594c53151b4d00a1784b6366e2814a (DEFAULT_TABLE table=rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.196039 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f594c53151b4d00a1784b6366e2814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.196313 11623 tablet_service.cc:1467] Processing CreateTablet for tablet b2ecce0ca7a941dbb98b89b797beb8a1 (DEFAULT_TABLE table=rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.196571 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ecce0ca7a941dbb98b89b797beb8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.196816 11622 tablet_service.cc:1467] Processing CreateTablet for tablet e6c1d3814e364e718449c3e4379b31ed (DEFAULT_TABLE table=rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.197019 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c1d3814e364e718449c3e4379b31ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.198810 13309 tablet_bootstrap.cc:492] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:30.199795 13309 tablet_bootstrap.cc:654] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.200273 11492 tablet_service.cc:1467] Processing CreateTablet for tablet b2ecce0ca7a941dbb98b89b797beb8a1 (DEFAULT_TABLE table=rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.200510 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ecce0ca7a941dbb98b89b797beb8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.200879 13307 tablet_bootstrap.cc:492] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:30.201349 13308 tablet_bootstrap.cc:492] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:30.201843 13307 tablet_bootstrap.cc:654] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.202219 13308 tablet_bootstrap.cc:654] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.202685 11491 tablet_service.cc:1467] Processing CreateTablet for tablet e6c1d3814e364e718449c3e4379b31ed (DEFAULT_TABLE table=rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.202919 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c1d3814e364e718449c3e4379b31ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.204016 13307 tablet_bootstrap.cc:492] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:30.204100 13307 ts_tablet_manager.cc:1367] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:30.204257 13308 tablet_bootstrap.cc:492] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:30.204327 13308 ts_tablet_manager.cc:1367] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 19:56:30.204558 13307 raft_consensus.cc:348] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.204663 13307 raft_consensus.cc:374] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.204692 13307 raft_consensus.cc:729] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.204708 13308 raft_consensus.cc:348] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.204798 13308 raft_consensus.cc:374] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.204826 13308 raft_consensus.cc:729] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.204811 13307 consensus_queue.cc:260] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.204923 13308 consensus_queue.cc:260] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.205085 13307 ts_tablet_manager.cc:1398] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:30.205042 11228 tablet_service.cc:1467] Processing CreateTablet for tablet b2ecce0ca7a941dbb98b89b797beb8a1 (DEFAULT_TABLE table=rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.205164 13308 ts_tablet_manager.cc:1398] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:30.205248 13307 tablet_bootstrap.cc:492] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:30.205273 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ecce0ca7a941dbb98b89b797beb8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.206105 13307 tablet_bootstrap.cc:654] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.207216 13307 tablet_bootstrap.cc:492] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:30.207299 13307 ts_tablet_manager.cc:1367] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:30.207549 13308 tablet_bootstrap.cc:492] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:30.207798 13307 raft_consensus.cc:348] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:30.207890 13307 raft_consensus.cc:374] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.207916 13307 raft_consensus.cc:729] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.208029 13307 consensus_queue.cc:260] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:30.208271 13307 ts_tablet_manager.cc:1398] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:30.208415 13308 tablet_bootstrap.cc:654] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.209599 13308 tablet_bootstrap.cc:492] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:30.209681 13308 ts_tablet_manager.cc:1367] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:30.210090 13308 raft_consensus.cc:348] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:30.210193 13308 raft_consensus.cc:374] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.210222 13308 raft_consensus.cc:729] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.210340 13308 consensus_queue.cc:260] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:30.210451 13309 tablet_bootstrap.cc:492] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:30.210547 13309 ts_tablet_manager.cc:1367] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:30.210595 13308 ts_tablet_manager.cc:1398] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:30.210939 13309 raft_consensus.cc:348] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:30.211031 13309 raft_consensus.cc:374] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.211058 13309 raft_consensus.cc:729] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.211210 13309 consensus_queue.cc:260] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:30.212666 13309 ts_tablet_manager.cc:1398] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:30.212909 13309 tablet_bootstrap.cc:492] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:30.213869 13309 tablet_bootstrap.cc:654] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.210085 13310 tablet_bootstrap.cc:492] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:30.215685 13310 tablet_bootstrap.cc:654] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.216511 13309 tablet_bootstrap.cc:492] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:30.216598 13309 ts_tablet_manager.cc:1367] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:30.216745 13310 tablet_bootstrap.cc:492] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:30.216866 13310 ts_tablet_manager.cc:1367] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:30.217125 13309 raft_consensus.cc:348] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:30.217298 13309 raft_consensus.cc:374] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.217372 13309 raft_consensus.cc:729] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.217334 13310 raft_consensus.cc:348] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.217454 13310 raft_consensus.cc:374] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.217518 13310 raft_consensus.cc:729] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.217556 13309 consensus_queue.cc:260] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:30.217649 13310 consensus_queue.cc:260] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.217969 13310 ts_tablet_manager.cc:1398] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:30.218183 13310 tablet_bootstrap.cc:492] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:30.218482 13309 ts_tablet_manager.cc:1398] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:30.219040 13310 tablet_bootstrap.cc:654] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.220260 13310 tablet_bootstrap.cc:492] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:30.220351 13310 ts_tablet_manager.cc:1367] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:30.220737 13310 raft_consensus.cc:348] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:30.220847 13310 raft_consensus.cc:374] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.220878 13310 raft_consensus.cc:729] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.221019 13301 raft_consensus.cc:482] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:30.221082 13301 raft_consensus.cc:504] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.221418 13301 leader_election.cc:288] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:30.221742 13310 consensus_queue.cc:260] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:30.222028 13310 ts_tablet_manager.cc:1398] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:30.222033 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f594c53151b4d00a1784b6366e2814a" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:30.222152 13310 tablet_bootstrap.cc:492] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:30.222185 11378 raft_consensus.cc:2454] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:30.222494 11447 leader_election.cc:302] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:30.222657 13301 raft_consensus.cc:2781] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:30.222707 13301 raft_consensus.cc:482] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:30.222749 13301 raft_consensus.cc:3037] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.222944 13310 tablet_bootstrap.cc:654] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.222910 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f594c53151b4d00a1784b6366e2814a" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:30.223043 11249 raft_consensus.cc:2454] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:30.223778 13301 raft_consensus.cc:504] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.224081 13301 leader_election.cc:288] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:30.224372 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f594c53151b4d00a1784b6366e2814a" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:30.224439 13310 tablet_bootstrap.cc:492] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:30.224496 13310 ts_tablet_manager.cc:1367] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:30.224484 11249 raft_consensus.cc:3037] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.224460 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f594c53151b4d00a1784b6366e2814a" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:30.224583 11378 raft_consensus.cc:3037] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.224797 13310 raft_consensus.cc:348] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:30.224859 13310 raft_consensus.cc:374] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.224877 13310 raft_consensus.cc:729] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.224980 13310 consensus_queue.cc:260] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:30.225207 13310 ts_tablet_manager.cc:1398] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:30.225406 11378 raft_consensus.cc:2454] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:30.225405 11249 raft_consensus.cc:2454] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:30.225716 11447 leader_election.cc:302] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:30.225875 13301 raft_consensus.cc:2781] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:30.225955 13301 raft_consensus.cc:686] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:30.226078 13301 consensus_queue.cc:237] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.228602 11100 catalog_manager.cc:5484] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:30.349133 13209 raft_consensus.cc:482] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:30.349249 13209 raft_consensus.cc:504] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:30.349627 13209 leader_election.cc:288] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:30.349995 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c1d3814e364e718449c3e4379b31ed" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:30.350054 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c1d3814e364e718449c3e4379b31ed" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:30.350152 11643 raft_consensus.cc:2454] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:30.350178 11508 raft_consensus.cc:2454] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:30.350443 11314 leader_election.cc:302] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:30.350658 13209 raft_consensus.cc:2781] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:30.350706 13209 raft_consensus.cc:482] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:30.350731 13209 raft_consensus.cc:3037] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.351567 13209 raft_consensus.cc:504] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:30.351897 13209 leader_election.cc:288] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:30.352176 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c1d3814e364e718449c3e4379b31ed" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:30.352283 11643 raft_consensus.cc:3037] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.352316 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c1d3814e364e718449c3e4379b31ed" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:30.352437 11508 raft_consensus.cc:3037] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.353082 11643 raft_consensus.cc:2454] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:30.353364 11314 leader_election.cc:302] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:30.353382 11508 raft_consensus.cc:2454] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:30.353525 13209 raft_consensus.cc:2781] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:30.353600 13209 raft_consensus.cc:686] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:30.353726 13209 consensus_queue.cc:237] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:30.355901 11100 catalog_manager.cc:5484] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:30.462363 13150 raft_consensus.cc:482] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:30.462505 13150 raft_consensus.cc:504] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:30.463200 13150 leader_election.cc:288] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:30.463681 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ecce0ca7a941dbb98b89b797beb8a1" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:30.463687 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ecce0ca7a941dbb98b89b797beb8a1" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:30.463825 11249 raft_consensus.cc:2454] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:30.463858 11512 raft_consensus.cc:2454] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:30.464169 11577 leader_election.cc:302] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:30.464339 13150 raft_consensus.cc:2781] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:30.464402 13150 raft_consensus.cc:482] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:30.464450 13150 raft_consensus.cc:3037] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.465380 13150 raft_consensus.cc:504] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:30.465724 13150 leader_election.cc:288] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:30.466084 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ecce0ca7a941dbb98b89b797beb8a1" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:30.466136 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ecce0ca7a941dbb98b89b797beb8a1" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:30.466202 11512 raft_consensus.cc:3037] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.466241 11249 raft_consensus.cc:3037] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.466964 11512 raft_consensus.cc:2454] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:30.467042 11249 raft_consensus.cc:2454] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:30.467353 11577 leader_election.cc:302] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:30.467645 13150 raft_consensus.cc:2781] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:30.467742 13150 raft_consensus.cc:686] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:30.467892 13150 consensus_queue.cc:237] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:30.469923 11100 catalog_manager.cc:5484] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:30.609056 13323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:30.609285 13323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:30.612303 13323 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240501 19:56:30.691944 11249 raft_consensus.cc:1261] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.692545 13150 consensus_queue.cc:1035] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:30.692734 11512 raft_consensus.cc:1261] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.693290 11381 raft_consensus.cc:1261] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.693727 13301 consensus_queue.cc:1035] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:30.694705 13150 consensus_queue.cc:1035] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:30.697623 11508 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.698438 12835 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:30.699296 11643 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.700074 13209 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:30.700510 11249 raft_consensus.cc:1261] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:30.701969 13301 consensus_queue.cc:1035] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.195 ms
time per row: 0.0610976 ms
I20240501 19:56:30.776083 11100 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:30.781729 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc0294d0dc5427e80411be5db9c95c1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.782086 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc0294d0dc5427e80411be5db9c95c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.782325 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc0294d0dc5427e80411be5db9c95c1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.782600 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc0294d0dc5427e80411be5db9c95c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.782624 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc0294d0dc5427e80411be5db9c95c1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.782857 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc0294d0dc5427e80411be5db9c95c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.782886 11492 tablet_service.cc:1467] Processing CreateTablet for tablet ea873ec8ced340b4ac082417c9b09aae (DEFAULT_TABLE table=rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.783075 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea873ec8ced340b4ac082417c9b09aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.784205 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 64a03f21e05f4999843f48cb4ebdb63c (DEFAULT_TABLE table=rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.784554 11227 tablet_service.cc:1467] Processing CreateTablet for tablet ea873ec8ced340b4ac082417c9b09aae (DEFAULT_TABLE table=rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.784926 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 64a03f21e05f4999843f48cb4ebdb63c (DEFAULT_TABLE table=rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.785151 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a03f21e05f4999843f48cb4ebdb63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.785753 11622 tablet_service.cc:1467] Processing CreateTablet for tablet 64a03f21e05f4999843f48cb4ebdb63c (DEFAULT_TABLE table=rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.785948 13387 tablet_bootstrap.cc:492] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:30.786011 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a03f21e05f4999843f48cb4ebdb63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.786326 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea873ec8ced340b4ac082417c9b09aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.786636 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a03f21e05f4999843f48cb4ebdb63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.786909 13387 tablet_bootstrap.cc:654] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.788029 13387 tablet_bootstrap.cc:492] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:30.788101 13387 ts_tablet_manager.cc:1367] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:30.788125 13391 tablet_bootstrap.cc:492] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:30.788228 11623 tablet_service.cc:1467] Processing CreateTablet for tablet ea873ec8ced340b4ac082417c9b09aae (DEFAULT_TABLE table=rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:30.788434 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea873ec8ced340b4ac082417c9b09aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:30.788520 13387 raft_consensus.cc:348] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.788614 13387 raft_consensus.cc:374] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.788643 13387 raft_consensus.cc:729] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.788759 13387 consensus_queue.cc:260] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.789034 13387 ts_tablet_manager.cc:1398] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:30.789175 13387 tablet_bootstrap.cc:492] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:30.789948 13387 tablet_bootstrap.cc:654] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.790076 13388 tablet_bootstrap.cc:492] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:30.790948 13388 tablet_bootstrap.cc:654] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.791307 13389 tablet_bootstrap.cc:492] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:30.791990 13387 tablet_bootstrap.cc:492] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:30.792003 13388 tablet_bootstrap.cc:492] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:30.792066 13387 ts_tablet_manager.cc:1367] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:30.792079 13388 ts_tablet_manager.cc:1367] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:30.792146 13389 tablet_bootstrap.cc:654] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.792469 13387 raft_consensus.cc:348] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.792557 13387 raft_consensus.cc:374] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.792585 13387 raft_consensus.cc:729] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.792513 13388 raft_consensus.cc:348] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.792621 13388 raft_consensus.cc:374] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.792649 13388 raft_consensus.cc:729] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.792694 13387 consensus_queue.cc:260] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.792765 13388 consensus_queue.cc:260] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.792951 13387 ts_tablet_manager.cc:1398] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:30.793036 13388 ts_tablet_manager.cc:1398] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:30.793195 13388 tablet_bootstrap.cc:492] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:30.793318 13391 tablet_bootstrap.cc:654] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.793984 13388 tablet_bootstrap.cc:654] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.794739 13389 tablet_bootstrap.cc:492] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:30.794831 13389 ts_tablet_manager.cc:1367] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:30.795258 13389 raft_consensus.cc:348] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.795365 13389 raft_consensus.cc:374] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.796270 13388 tablet_bootstrap.cc:492] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:30.796339 13391 tablet_bootstrap.cc:492] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:30.796346 13388 ts_tablet_manager.cc:1367] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:30.796437 13391 ts_tablet_manager.cc:1367] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:30.795408 13389 raft_consensus.cc:729] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.796588 13389 consensus_queue.cc:260] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.796747 13388 raft_consensus.cc:348] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.796845 13388 raft_consensus.cc:374] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.796865 13389 ts_tablet_manager.cc:1398] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:30.796876 13388 raft_consensus.cc:729] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.796836 13391 raft_consensus.cc:348] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.796948 13391 raft_consensus.cc:374] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.796988 13391 raft_consensus.cc:729] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.796999 13389 tablet_bootstrap.cc:492] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:30.796972 13388 consensus_queue.cc:260] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.797103 13391 consensus_queue.cc:260] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.797366 13391 ts_tablet_manager.cc:1398] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:30.797535 13391 tablet_bootstrap.cc:492] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:30.797740 13389 tablet_bootstrap.cc:654] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.798215 13388 ts_tablet_manager.cc:1398] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:30.798355 13391 tablet_bootstrap.cc:654] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.799012 13389 tablet_bootstrap.cc:492] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:30.799094 13389 ts_tablet_manager.cc:1367] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:30.799527 13389 raft_consensus.cc:348] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.799633 13389 raft_consensus.cc:374] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.799660 13389 raft_consensus.cc:729] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.799762 13389 consensus_queue.cc:260] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.800073 13389 ts_tablet_manager.cc:1398] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:30.800254 13389 tablet_bootstrap.cc:492] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:30.801087 13391 tablet_bootstrap.cc:492] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:30.801131 13389 tablet_bootstrap.cc:654] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:30.801167 13391 ts_tablet_manager.cc:1367] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:30.802120 13389 tablet_bootstrap.cc:492] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:30.802197 13389 ts_tablet_manager.cc:1367] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:30.801579 13391 raft_consensus.cc:348] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.802311 13391 raft_consensus.cc:374] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.802362 13391 raft_consensus.cc:729] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.802479 13391 consensus_queue.cc:260] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.802639 13389 raft_consensus.cc:348] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.802736 13389 raft_consensus.cc:374] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:30.802768 13389 raft_consensus.cc:729] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:30.802796 13391 ts_tablet_manager.cc:1398] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:30.802875 13389 consensus_queue.cc:260] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.803130 13389 ts_tablet_manager.cc:1398] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:30.837723 13301 raft_consensus.cc:482] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:30.837862 13301 raft_consensus.cc:504] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.838299 13301 leader_election.cc:288] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:30.838757 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc0294d0dc5427e80411be5db9c95c1" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:30.838758 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc0294d0dc5427e80411be5db9c95c1" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:30.838891 11248 raft_consensus.cc:2454] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:30.838904 11381 raft_consensus.cc:2454] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:30.839203 11447 leader_election.cc:302] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:30.839403 13301 raft_consensus.cc:2781] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:30.839505 13301 raft_consensus.cc:482] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:30.839551 13301 raft_consensus.cc:3037] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.840507 13301 raft_consensus.cc:504] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.840726 13220 raft_consensus.cc:482] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:30.840822 13301 leader_election.cc:288] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:30.840826 13220 raft_consensus.cc:504] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.841130 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc0294d0dc5427e80411be5db9c95c1" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:30.841200 13220 leader_election.cc:288] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:30.841244 11381 raft_consensus.cc:3037] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.841244 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc0294d0dc5427e80411be5db9c95c1" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:30.841360 11248 raft_consensus.cc:3037] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.841526 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a03f21e05f4999843f48cb4ebdb63c" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:30.841637 11378 raft_consensus.cc:2454] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:30.841897 11183 leader_election.cc:302] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:30.842056 13220 raft_consensus.cc:2781] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:30.842101 13220 raft_consensus.cc:482] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:30.842147 13220 raft_consensus.cc:3037] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.842170 11381 raft_consensus.cc:2454] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:30.842336 11248 raft_consensus.cc:2454] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:30.842453 11447 leader_election.cc:302] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:30.842660 13301 raft_consensus.cc:2781] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:30.842746 13301 raft_consensus.cc:686] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:30.842871 13301 consensus_queue.cc:237] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.842988 13220 raft_consensus.cc:504] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.843276 13220 leader_election.cc:288] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:30.843405 13155 raft_consensus.cc:482] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:30.843531 13155 raft_consensus.cc:504] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.843763 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a03f21e05f4999843f48cb4ebdb63c" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:30.843770 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a03f21e05f4999843f48cb4ebdb63c" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:30.843827 13155 leader_election.cc:288] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:30.843856 11381 raft_consensus.cc:3037] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.843889 11643 raft_consensus.cc:3037] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.844467 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea873ec8ced340b4ac082417c9b09aae" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:30.844574 11248 raft_consensus.cc:2454] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:30.844723 11381 raft_consensus.cc:2454] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:30.844794 11643 raft_consensus.cc:2454] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:30.844873 11445 leader_election.cc:302] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:30.845008 13155 raft_consensus.cc:2781] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:30.845011 11183 leader_election.cc:302] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:30.845055 13155 raft_consensus.cc:482] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:30.845089 13155 raft_consensus.cc:3037] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.845244 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a03f21e05f4999843f48cb4ebdb63c" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:30.845258 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea873ec8ced340b4ac082417c9b09aae" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:30.845366 11644 raft_consensus.cc:2362] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1. Re-sending same reply.
I20240501 19:56:30.845383 11643 raft_consensus.cc:2454] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:30.845858 13220 raft_consensus.cc:2781] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:30.845971 13220 raft_consensus.cc:686] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:30.845988 13155 raft_consensus.cc:504] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.845928 11102 catalog_manager.cc:5484] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:30.846096 13220 consensus_queue.cc:237] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:30.846287 13155 leader_election.cc:288] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:30.846614 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea873ec8ced340b4ac082417c9b09aae" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:30.846702 11643 raft_consensus.cc:3037] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.847520 11643 raft_consensus.cc:2454] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:30.847534 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea873ec8ced340b4ac082417c9b09aae" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:30.847615 11248 raft_consensus.cc:3037] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:30.847774 11446 leader_election.cc:302] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:30.847927 13155 raft_consensus.cc:2781] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:30.848001 13155 raft_consensus.cc:686] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:30.848109 13155 consensus_queue.cc:237] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:30.848407 11248 raft_consensus.cc:2454] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:30.848644 11105 catalog_manager.cc:5484] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:30.850533 11102 catalog_manager.cc:5484] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:30.965866 13403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:30.966110 13403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:30.969365 13403 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240501 19:56:31.058486 11643 raft_consensus.cc:1261] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.058490 11381 raft_consensus.cc:1261] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.058972 13218 consensus_queue.cc:1035] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.059566 13218 consensus_queue.cc:1035] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.061671 11248 raft_consensus.cc:1261] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.062206 13155 consensus_queue.cc:1035] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.066452 11644 raft_consensus.cc:1261] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.070715 13155 consensus_queue.cc:1035] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.096726 11248 raft_consensus.cc:1261] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.097223 13301 consensus_queue.cc:1035] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.098701 11381 raft_consensus.cc:1261] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.099053 13437 consensus_queue.cc:1035] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.109475 13359 mvcc.cc:205] Tried to move back new op lower bound from 7022974529771855872 to 7022974528893853696. Current Snapshot: MvccSnapshot[applied={T|T < 7022974529771855872}]
INSERT report
rows total: 2000
time total: 120.874 ms
time per row: 0.0604371 ms
I20240501 19:56:31.131929 11104 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:31.137290 11492 tablet_service.cc:1467] Processing CreateTablet for tablet cebd15701ee44b11a2f8804740fb4299 (DEFAULT_TABLE table=rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.137617 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cebd15701ee44b11a2f8804740fb4299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.137866 11622 tablet_service.cc:1467] Processing CreateTablet for tablet a3429e6c8f1449d29bb7b9c1c6a0bb49 (DEFAULT_TABLE table=rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.138116 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3429e6c8f1449d29bb7b9c1c6a0bb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.138150 11490 tablet_service.cc:1467] Processing CreateTablet for tablet 1c07d9fc05254a6399afcc54b3efb8ac (DEFAULT_TABLE table=rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.138398 11490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c07d9fc05254a6399afcc54b3efb8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.138569 11360 tablet_service.cc:1467] Processing CreateTablet for tablet 1c07d9fc05254a6399afcc54b3efb8ac (DEFAULT_TABLE table=rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.138820 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c07d9fc05254a6399afcc54b3efb8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.140009 11623 tablet_service.cc:1467] Processing CreateTablet for tablet cebd15701ee44b11a2f8804740fb4299 (DEFAULT_TABLE table=rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.140221 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cebd15701ee44b11a2f8804740fb4299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.140290 11491 tablet_service.cc:1467] Processing CreateTablet for tablet a3429e6c8f1449d29bb7b9c1c6a0bb49 (DEFAULT_TABLE table=rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.140494 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3429e6c8f1449d29bb7b9c1c6a0bb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.142349 11359 tablet_service.cc:1467] Processing CreateTablet for tablet a3429e6c8f1449d29bb7b9c1c6a0bb49 (DEFAULT_TABLE table=rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.142410 11358 tablet_service.cc:1467] Processing CreateTablet for tablet cebd15701ee44b11a2f8804740fb4299 (DEFAULT_TABLE table=rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.142596 13391 tablet_bootstrap.cc:492] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:31.142616 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3429e6c8f1449d29bb7b9c1c6a0bb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.142742 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cebd15701ee44b11a2f8804740fb4299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.142916 13388 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:31.143573 13387 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:31.143720 13388 tablet_bootstrap.cc:654] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.144436 13387 tablet_bootstrap.cc:654] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.145399 13391 tablet_bootstrap.cc:654] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.145743 13387 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:31.145810 13387 ts_tablet_manager.cc:1367] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:31.146298 13387 raft_consensus.cc:348] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.146410 13387 raft_consensus.cc:374] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.146441 13387 raft_consensus.cc:729] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.146553 13387 consensus_queue.cc:260] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.146610 13391 tablet_bootstrap.cc:492] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:31.146687 13391 ts_tablet_manager.cc:1367] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:31.146811 13387 ts_tablet_manager.cc:1398] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:31.146773 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 1c07d9fc05254a6399afcc54b3efb8ac (DEFAULT_TABLE table=rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.146989 13387 tablet_bootstrap.cc:492] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:31.147033 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c07d9fc05254a6399afcc54b3efb8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.147224 13391 raft_consensus.cc:348] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.147312 13391 raft_consensus.cc:374] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.147341 13391 raft_consensus.cc:729] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.147825 13387 tablet_bootstrap.cc:654] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.148535 13391 consensus_queue.cc:260] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.148826 13391 ts_tablet_manager.cc:1398] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:31.148988 13391 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:31.149178 13389 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:31.149884 13391 tablet_bootstrap.cc:654] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.149912 13389 tablet_bootstrap.cc:654] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.150768 13387 tablet_bootstrap.cc:492] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:31.150839 13387 ts_tablet_manager.cc:1367] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:31.151285 13387 raft_consensus.cc:348] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.151402 13387 raft_consensus.cc:374] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.151476 13387 raft_consensus.cc:729] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.151590 13387 consensus_queue.cc:260] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.151866 13387 ts_tablet_manager.cc:1398] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:31.152098 13387 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:31.153167 13387 tablet_bootstrap.cc:654] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.153216 13389 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:31.153311 13389 ts_tablet_manager.cc:1367] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:31.154004 13389 raft_consensus.cc:348] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.154114 13389 raft_consensus.cc:374] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.154145 13389 raft_consensus.cc:729] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.154255 13389 consensus_queue.cc:260] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.154299 13387 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:31.154371 13387 ts_tablet_manager.cc:1367] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:31.154382 13391 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:31.154456 13391 ts_tablet_manager.cc:1367] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 19:56:31.154515 13389 ts_tablet_manager.cc:1398] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.154683 13387 raft_consensus.cc:348] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.154762 13387 raft_consensus.cc:374] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.154789 13387 raft_consensus.cc:729] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.154861 13391 raft_consensus.cc:348] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.154918 13387 consensus_queue.cc:260] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.154975 13391 raft_consensus.cc:374] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.155006 13391 raft_consensus.cc:729] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.155144 13391 consensus_queue.cc:260] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.155244 13387 ts_tablet_manager.cc:1398] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:31.155503 13391 ts_tablet_manager.cc:1398] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:31.156320 13388 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:31.156409 13388 ts_tablet_manager.cc:1367] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:56:31.156833 13388 raft_consensus.cc:348] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.156948 13388 raft_consensus.cc:374] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.156976 13388 raft_consensus.cc:729] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.157084 13388 consensus_queue.cc:260] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.157316 13388 ts_tablet_manager.cc:1398] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:31.157460 13388 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:31.158227 13388 tablet_bootstrap.cc:654] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.160072 13388 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:31.160162 13388 ts_tablet_manager.cc:1367] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:31.160581 13388 raft_consensus.cc:348] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.160804 13388 raft_consensus.cc:374] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.160851 13388 raft_consensus.cc:729] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.160979 13388 consensus_queue.cc:260] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.161288 13388 ts_tablet_manager.cc:1398] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:31.161513 13388 tablet_bootstrap.cc:492] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:31.162389 13388 tablet_bootstrap.cc:654] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.163400 13388 tablet_bootstrap.cc:492] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:31.163491 13388 ts_tablet_manager.cc:1367] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:31.163887 13388 raft_consensus.cc:348] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.164002 13388 raft_consensus.cc:374] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.164048 13388 raft_consensus.cc:729] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.164168 13388 consensus_queue.cc:260] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.164458 13388 ts_tablet_manager.cc:1398] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:31.207325 13155 raft_consensus.cc:482] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:31.207475 13155 raft_consensus.cc:504] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.207829 13155 leader_election.cc:288] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.208184 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3429e6c8f1449d29bb7b9c1c6a0bb49" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:31.208200 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3429e6c8f1449d29bb7b9c1c6a0bb49" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:31.208307 11643 raft_consensus.cc:2454] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:31.208350 11378 raft_consensus.cc:2454] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:31.208581 11446 leader_election.cc:302] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:31.209013 13155 raft_consensus.cc:2781] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:31.209066 13155 raft_consensus.cc:482] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:31.209100 13155 raft_consensus.cc:3037] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.209928 13155 raft_consensus.cc:504] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.210213 13155 leader_election.cc:288] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.210494 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3429e6c8f1449d29bb7b9c1c6a0bb49" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:31.210600 11378 raft_consensus.cc:3037] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.210603 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3429e6c8f1449d29bb7b9c1c6a0bb49" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:31.210732 11643 raft_consensus.cc:3037] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.211515 11378 raft_consensus.cc:2454] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:31.211711 11643 raft_consensus.cc:2454] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:31.211797 11447 leader_election.cc:302] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:31.211970 13155 raft_consensus.cc:2781] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:31.212069 13155 raft_consensus.cc:686] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:31.212193 13155 consensus_queue.cc:237] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.213899 11104 catalog_manager.cc:5484] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:31.244875 13386 raft_consensus.cc:482] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:31.245020 13386 raft_consensus.cc:504] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.245479 13386 leader_election.cc:288] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.245841 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cebd15701ee44b11a2f8804740fb4299" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:31.245877 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cebd15701ee44b11a2f8804740fb4299" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:31.245996 11643 raft_consensus.cc:2454] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:31.246034 11512 raft_consensus.cc:2454] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:31.246286 11314 leader_election.cc:302] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:31.246474 13386 raft_consensus.cc:2781] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:31.246533 13386 raft_consensus.cc:482] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:31.246580 13386 raft_consensus.cc:3037] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.247468 13386 raft_consensus.cc:504] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.247783 13386 leader_election.cc:288] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.248108 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cebd15701ee44b11a2f8804740fb4299" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:31.248099 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cebd15701ee44b11a2f8804740fb4299" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:31.248222 11512 raft_consensus.cc:3037] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.248221 11643 raft_consensus.cc:3037] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.249171 11643 raft_consensus.cc:2454] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:31.249171 11512 raft_consensus.cc:2454] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:31.249476 11314 leader_election.cc:302] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:31.249652 13386 raft_consensus.cc:2781] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:31.249745 13386 raft_consensus.cc:686] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:31.249902 13386 consensus_queue.cc:237] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.251849 11104 catalog_manager.cc:5484] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:31.259184 13434 raft_consensus.cc:482] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:31.259294 13434 raft_consensus.cc:504] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.259701 13434 leader_election.cc:288] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:31.260032 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c07d9fc05254a6399afcc54b3efb8ac" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:31.260082 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c07d9fc05254a6399afcc54b3efb8ac" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:31.260165 11378 raft_consensus.cc:2454] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:31.260191 11512 raft_consensus.cc:2454] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:31.260454 11183 leader_election.cc:302] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:31.260612 13434 raft_consensus.cc:2781] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:31.260668 13434 raft_consensus.cc:482] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:31.260712 13434 raft_consensus.cc:3037] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.261461 13434 raft_consensus.cc:504] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.261787 13434 leader_election.cc:288] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:31.262081 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c07d9fc05254a6399afcc54b3efb8ac" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:31.262100 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c07d9fc05254a6399afcc54b3efb8ac" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:31.262180 11378 raft_consensus.cc:3037] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.262212 11512 raft_consensus.cc:3037] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.264356 11378 raft_consensus.cc:2454] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:31.264593 11512 raft_consensus.cc:2454] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:31.264935 11183 leader_election.cc:302] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:31.265270 13434 raft_consensus.cc:2781] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:31.265385 13434 raft_consensus.cc:686] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:31.265511 13434 consensus_queue.cc:237] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.268054 11104 catalog_manager.cc:5484] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:31.414237 13452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:31.414484 13452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:31.417639 13452 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240501 19:56:31.506640 11508 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.507056 13209 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.507714 11643 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.508935 11378 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.509132 13209 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.509414 11643 raft_consensus.cc:1261] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.509402 13434 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.509783 13155 consensus_queue.cc:1035] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.510149 11508 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.510515 11378 raft_consensus.cc:1261] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.514843 13301 consensus_queue.cc:1035] T a3429e6c8f1449d29bb7b9c1c6a0bb49 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.515281 13434 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.539289 13358 mvcc.cc:205] Tried to move back new op lower bound from 7022974531601534976 to 7022974530561085440. Current Snapshot: MvccSnapshot[applied={T|T < 7022974531601534976 or (T in {7022974531601534976})}]
I20240501 19:56:31.570580 13355 mvcc.cc:205] Tried to move back new op lower bound from 7022974531618779136 to 7022974530406264832. Current Snapshot: MvccSnapshot[applied={T|T < 7022974531618779136}]
INSERT report
rows total: 2000
time total: 121.337 ms
time per row: 0.0606686 ms
I20240501 19:56:31.582635 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:31.588311 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 01ed8143e5454a4aaef3e142069f35a2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.588606 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ed8143e5454a4aaef3e142069f35a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.588553 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 01ed8143e5454a4aaef3e142069f35a2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.588833 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ed8143e5454a4aaef3e142069f35a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.590538 13387 tablet_bootstrap.cc:492] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:31.590670 13389 tablet_bootstrap.cc:492] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:31.591315 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 01ed8143e5454a4aaef3e142069f35a2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.591428 13387 tablet_bootstrap.cc:654] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.591429 13389 tablet_bootstrap.cc:654] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.591581 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ed8143e5454a4aaef3e142069f35a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.593295 11491 tablet_service.cc:1467] Processing CreateTablet for tablet b7073666bc3f4c0a8c1e003fb1ef4fef (DEFAULT_TABLE table=rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.593549 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7073666bc3f4c0a8c1e003fb1ef4fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.593531 11227 tablet_service.cc:1467] Processing CreateTablet for tablet b7073666bc3f4c0a8c1e003fb1ef4fef (DEFAULT_TABLE table=rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.593636 11622 tablet_service.cc:1467] Processing CreateTablet for tablet b7073666bc3f4c0a8c1e003fb1ef4fef (DEFAULT_TABLE table=rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.593531 11229 tablet_service.cc:1467] Processing CreateTablet for tablet c31cf707a7e84f139fce27df22b376fd (DEFAULT_TABLE table=rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.593765 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7073666bc3f4c0a8c1e003fb1ef4fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.593843 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31cf707a7e84f139fce27df22b376fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.593832 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7073666bc3f4c0a8c1e003fb1ef4fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.595563 13391 tablet_bootstrap.cc:492] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:31.595635 11624 tablet_service.cc:1467] Processing CreateTablet for tablet c31cf707a7e84f139fce27df22b376fd (DEFAULT_TABLE table=rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.595813 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31cf707a7e84f139fce27df22b376fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.596349 13391 tablet_bootstrap.cc:654] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.596415 11358 tablet_service.cc:1467] Processing CreateTablet for tablet c31cf707a7e84f139fce27df22b376fd (DEFAULT_TABLE table=rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.596594 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31cf707a7e84f139fce27df22b376fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.596927 13388 tablet_bootstrap.cc:492] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:31.597733 13388 tablet_bootstrap.cc:654] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.598567 13389 tablet_bootstrap.cc:492] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:31.598591 13391 tablet_bootstrap.cc:492] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:31.598650 13389 ts_tablet_manager.cc:1367] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:31.598663 13391 ts_tablet_manager.cc:1367] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:31.598686 13388 tablet_bootstrap.cc:492] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:31.598753 13388 ts_tablet_manager.cc:1367] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:31.599073 13391 raft_consensus.cc:348] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.599103 13389 raft_consensus.cc:348] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.599172 13391 raft_consensus.cc:374] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.599131 13388 raft_consensus.cc:348] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.599200 13389 raft_consensus.cc:374] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.599201 13391 raft_consensus.cc:729] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.599210 13388 raft_consensus.cc:374] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.599231 13389 raft_consensus.cc:729] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.599241 13388 raft_consensus.cc:729] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.599330 13388 consensus_queue.cc:260] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.599315 13391 consensus_queue.cc:260] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.599339 13389 consensus_queue.cc:260] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.599606 13391 ts_tablet_manager.cc:1398] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.599634 13389 ts_tablet_manager.cc:1398] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:31.599653 13388 ts_tablet_manager.cc:1398] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.599757 13391 tablet_bootstrap.cc:492] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:31.599756 13387 tablet_bootstrap.cc:492] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:31.599766 13389 tablet_bootstrap.cc:492] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:31.599833 13387 ts_tablet_manager.cc:1367] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:31.600258 13387 raft_consensus.cc:348] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.600351 13387 raft_consensus.cc:374] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.600380 13387 raft_consensus.cc:729] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.600517 13391 tablet_bootstrap.cc:654] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.600502 13387 consensus_queue.cc:260] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.600597 13389 tablet_bootstrap.cc:654] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.600787 13387 ts_tablet_manager.cc:1398] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:31.600939 13387 tablet_bootstrap.cc:492] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:31.601805 13387 tablet_bootstrap.cc:654] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.602036 13391 tablet_bootstrap.cc:492] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:31.602102 13391 ts_tablet_manager.cc:1367] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:31.602464 13391 raft_consensus.cc:348] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.602547 13391 raft_consensus.cc:374] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.602571 13391 raft_consensus.cc:729] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.602664 13391 consensus_queue.cc:260] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.602917 13391 ts_tablet_manager.cc:1398] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:31.603036 13391 tablet_bootstrap.cc:492] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:31.603808 13391 tablet_bootstrap.cc:654] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.604982 13391 tablet_bootstrap.cc:492] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:31.605053 13391 ts_tablet_manager.cc:1367] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:31.606331 13387 tablet_bootstrap.cc:492] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:31.606410 13387 ts_tablet_manager.cc:1367] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:31.606454 13391 raft_consensus.cc:348] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.606551 13391 raft_consensus.cc:374] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.606580 13391 raft_consensus.cc:729] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.606803 13391 consensus_queue.cc:260] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.606866 13387 raft_consensus.cc:348] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.606990 13387 raft_consensus.cc:374] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.607028 13387 raft_consensus.cc:729] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.607069 13389 tablet_bootstrap.cc:492] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:31.607110 13391 ts_tablet_manager.cc:1398] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:31.607146 13389 ts_tablet_manager.cc:1367] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:31.607136 13387 consensus_queue.cc:260] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.607396 13387 ts_tablet_manager.cc:1398] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.607578 13389 raft_consensus.cc:348] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.607671 13389 raft_consensus.cc:374] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.607697 13389 raft_consensus.cc:729] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.607797 13389 consensus_queue.cc:260] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.608028 13389 ts_tablet_manager.cc:1398] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.608158 13389 tablet_bootstrap.cc:492] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:31.608980 13389 tablet_bootstrap.cc:654] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.610133 13389 tablet_bootstrap.cc:492] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:31.610211 13389 ts_tablet_manager.cc:1367] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:31.610867 13209 raft_consensus.cc:482] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:31.610877 13389 raft_consensus.cc:348] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.610958 13389 raft_consensus.cc:374] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.610944 13209 raft_consensus.cc:504] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.610998 13389 raft_consensus.cc:729] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.611160 13389 consensus_queue.cc:260] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.611281 13209 leader_election.cc:288] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.611737 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ed8143e5454a4aaef3e142069f35a2" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:31.611840 13389 ts_tablet_manager.cc:1398] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:31.611860 11643 raft_consensus.cc:2454] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:31.612111 11314 leader_election.cc:302] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:31.612263 13209 raft_consensus.cc:2781] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:31.612318 13209 raft_consensus.cc:482] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:31.612354 13209 raft_consensus.cc:3037] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.612838 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ed8143e5454a4aaef3e142069f35a2" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:31.612932 11248 raft_consensus.cc:2454] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:31.613325 13209 raft_consensus.cc:504] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.613642 13209 leader_election.cc:288] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.614022 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ed8143e5454a4aaef3e142069f35a2" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:31.614101 11248 raft_consensus.cc:3037] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.614320 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ed8143e5454a4aaef3e142069f35a2" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:31.614429 11643 raft_consensus.cc:3037] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.615022 11248 raft_consensus.cc:2454] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:31.615274 11313 leader_election.cc:302] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:31.615280 11643 raft_consensus.cc:2454] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:31.615564 13209 raft_consensus.cc:2781] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:31.615640 13209 raft_consensus.cc:686] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:31.615757 13209 consensus_queue.cc:237] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:31.617626 11102 catalog_manager.cc:5484] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:31.670549 13301 raft_consensus.cc:482] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:31.670672 13301 raft_consensus.cc:504] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.671066 13301 leader_election.cc:288] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.671361 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7073666bc3f4c0a8c1e003fb1ef4fef" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:31.671525 11248 raft_consensus.cc:2454] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:31.671554 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7073666bc3f4c0a8c1e003fb1ef4fef" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:31.671707 11643 raft_consensus.cc:2454] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:31.671808 11445 leader_election.cc:302] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:31.671973 13301 raft_consensus.cc:2781] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:31.672015 13301 raft_consensus.cc:482] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:31.672050 13301 raft_consensus.cc:3037] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.672865 13301 raft_consensus.cc:504] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.673229 13301 leader_election.cc:288] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.673528 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7073666bc3f4c0a8c1e003fb1ef4fef" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:31.673569 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7073666bc3f4c0a8c1e003fb1ef4fef" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:31.673635 11248 raft_consensus.cc:3037] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.673661 11643 raft_consensus.cc:3037] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.674417 11248 raft_consensus.cc:2454] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:31.674437 11643 raft_consensus.cc:2454] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:31.674649 11446 leader_election.cc:302] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:31.674834 13301 raft_consensus.cc:2781] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:31.674906 13301 raft_consensus.cc:686] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:31.675035 13301 consensus_queue.cc:237] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.676990 11102 catalog_manager.cc:5484] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:31.683533 13218 raft_consensus.cc:482] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:31.683625 13218 raft_consensus.cc:504] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.683966 13218 leader_election.cc:288] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.684280 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31cf707a7e84f139fce27df22b376fd" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:31.684321 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31cf707a7e84f139fce27df22b376fd" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:31.684404 11643 raft_consensus.cc:2454] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:31.684454 11381 raft_consensus.cc:2454] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:31.684760 11183 leader_election.cc:302] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:31.684983 13218 raft_consensus.cc:2781] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:31.685042 13218 raft_consensus.cc:482] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:31.685089 13218 raft_consensus.cc:3037] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.685842 13218 raft_consensus.cc:504] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.686180 13218 leader_election.cc:288] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:31.686450 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31cf707a7e84f139fce27df22b376fd" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:31.686573 11381 raft_consensus.cc:3037] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.686580 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31cf707a7e84f139fce27df22b376fd" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:31.686691 11643 raft_consensus.cc:3037] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:31.687311 11381 raft_consensus.cc:2454] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:31.687412 11643 raft_consensus.cc:2454] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:31.687611 11183 leader_election.cc:302] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:31.687758 13218 raft_consensus.cc:2781] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:31.687837 13218 raft_consensus.cc:686] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:31.688001 13218 consensus_queue.cc:237] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.689879 11102 catalog_manager.cc:5484] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:31.819165 13503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:31.819460 13503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:31.822707 13503 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240501 19:56:31.901000 11381 raft_consensus.cc:1261] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.902951 13218 consensus_queue.cc:1035] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.903766 11643 raft_consensus.cc:1261] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.905107 11643 raft_consensus.cc:1261] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.907688 13386 consensus_queue.cc:1035] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.907773 11248 raft_consensus.cc:1261] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.908179 13386 consensus_queue.cc:1035] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.909906 13434 consensus_queue.cc:1035] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.913100 11249 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.913486 13437 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.926409 11643 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:31.926889 13437 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:31.934953 13363 mvcc.cc:205] Tried to move back new op lower bound from 7022974533308674048 to 7022974532302159872. Current Snapshot: MvccSnapshot[applied={T|T < 7022974533207326720}]
INSERT report
rows total: 2000
time total: 105.554 ms
time per row: 0.0527772 ms
I20240501 19:56:31.975368 11098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:31.981699 11227 tablet_service.cc:1467] Processing CreateTablet for tablet c3b35dab7eec4b73b208f687dad46ba2 (DEFAULT_TABLE table=rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.982026 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b35dab7eec4b73b208f687dad46ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.982211 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 1b57520d53734c318d603dd46e30b190 (DEFAULT_TABLE table=rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.982335 11491 tablet_service.cc:1467] Processing CreateTablet for tablet c3b35dab7eec4b73b208f687dad46ba2 (DEFAULT_TABLE table=rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.982502 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b57520d53734c318d603dd46e30b190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.982579 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b35dab7eec4b73b208f687dad46ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.983834 13389 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:31.984088 11358 tablet_service.cc:1467] Processing CreateTablet for tablet c3b35dab7eec4b73b208f687dad46ba2 (DEFAULT_TABLE table=rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.984349 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b35dab7eec4b73b208f687dad46ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.984601 13391 tablet_bootstrap.cc:492] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:31.984616 13388 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:31.984712 13389 tablet_bootstrap.cc:654] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.985407 13388 tablet_bootstrap.cc:654] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.985407 13391 tablet_bootstrap.cc:654] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.986320 13387 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:31.986558 13389 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:31.986639 13389 ts_tablet_manager.cc:1367] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:31.987080 13389 raft_consensus.cc:348] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.987164 13387 tablet_bootstrap.cc:654] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.987195 13389 raft_consensus.cc:374] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.987229 13389 raft_consensus.cc:729] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.987340 13389 consensus_queue.cc:260] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.987612 13388 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:31.987644 13389 ts_tablet_manager.cc:1398] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.987681 13388 ts_tablet_manager.cc:1367] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:31.987936 11490 tablet_service.cc:1467] Processing CreateTablet for tablet fe34db1ffe5f4509abb547f88c12a562 (DEFAULT_TABLE table=rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.988159 11490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe34db1ffe5f4509abb547f88c12a562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.988111 13388 raft_consensus.cc:348] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.988209 13388 raft_consensus.cc:374] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.988240 13388 raft_consensus.cc:729] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.988288 11358 tablet_service.cc:1467] Processing CreateTablet for tablet fe34db1ffe5f4509abb547f88c12a562 (DEFAULT_TABLE table=rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.988344 13388 consensus_queue.cc:260] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.988446 11624 tablet_service.cc:1467] Processing CreateTablet for tablet fe34db1ffe5f4509abb547f88c12a562 (DEFAULT_TABLE table=rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.988493 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe34db1ffe5f4509abb547f88c12a562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.988610 13388 ts_tablet_manager.cc:1398] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.988658 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe34db1ffe5f4509abb547f88c12a562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.989972 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 1b57520d53734c318d603dd46e30b190 (DEFAULT_TABLE table=rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.990186 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b57520d53734c318d603dd46e30b190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.990479 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 1b57520d53734c318d603dd46e30b190 (DEFAULT_TABLE table=rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:31.990707 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b57520d53734c318d603dd46e30b190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.990804 13388 tablet_bootstrap.cc:492] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:31.991091 13387 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:31.991158 13387 ts_tablet_manager.cc:1367] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:31.991668 13388 tablet_bootstrap.cc:654] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.991624 13387 raft_consensus.cc:348] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.991726 13387 raft_consensus.cc:374] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.991756 13387 raft_consensus.cc:729] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.991853 13387 consensus_queue.cc:260] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:31.992105 13387 ts_tablet_manager.cc:1398] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.992251 13387 tablet_bootstrap.cc:492] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:31.992887 13391 tablet_bootstrap.cc:492] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:31.992964 13391 ts_tablet_manager.cc:1367] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:31.993024 13387 tablet_bootstrap.cc:654] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.993384 13391 raft_consensus.cc:348] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.993475 13391 raft_consensus.cc:374] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.993502 13391 raft_consensus.cc:729] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.993606 13391 consensus_queue.cc:260] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.993857 13391 ts_tablet_manager.cc:1398] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.994004 13391 tablet_bootstrap.cc:492] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:31.994827 13391 tablet_bootstrap.cc:654] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:31.995188 13387 tablet_bootstrap.cc:492] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:31.995244 13387 ts_tablet_manager.cc:1367] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:31.995658 13387 raft_consensus.cc:348] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.995733 13387 raft_consensus.cc:374] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:31.995750 13387 raft_consensus.cc:729] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:31.995818 13387 consensus_queue.cc:260] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:31.996043 13387 ts_tablet_manager.cc:1398] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:31.998756 13389 tablet_bootstrap.cc:492] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:31.999702 13389 tablet_bootstrap.cc:654] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.000695 13391 tablet_bootstrap.cc:492] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:32.000751 13389 tablet_bootstrap.cc:492] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:32.000785 13391 ts_tablet_manager.cc:1367] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:32.000929 13389 ts_tablet_manager.cc:1367] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:32.001181 13391 raft_consensus.cc:348] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.001296 13391 raft_consensus.cc:374] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.001329 13391 raft_consensus.cc:729] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.001427 13389 raft_consensus.cc:348] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.001472 13391 consensus_queue.cc:260] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.001554 13389 raft_consensus.cc:374] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.001590 13389 raft_consensus.cc:729] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.001694 13389 consensus_queue.cc:260] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.002481 13391 ts_tablet_manager.cc:1398] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:32.002493 13389 ts_tablet_manager.cc:1398] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:32.003243 13388 tablet_bootstrap.cc:492] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:32.003366 13388 ts_tablet_manager.cc:1367] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 19:56:32.003922 13388 raft_consensus.cc:348] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.004014 13388 raft_consensus.cc:374] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.004045 13388 raft_consensus.cc:729] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.004187 13388 consensus_queue.cc:260] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.004530 13388 ts_tablet_manager.cc:1398] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:32.004738 13388 tablet_bootstrap.cc:492] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:32.005810 13388 tablet_bootstrap.cc:654] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:32.006749 13388 tablet_bootstrap.cc:492] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:32.006827 13388 ts_tablet_manager.cc:1367] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:32.007256 13388 raft_consensus.cc:348] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.007364 13388 raft_consensus.cc:374] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:32.007396 13388 raft_consensus.cc:729] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:32.007560 13388 consensus_queue.cc:260] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.007844 13388 ts_tablet_manager.cc:1398] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:32.037787 13301 raft_consensus.cc:482] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:32.037937 13301 raft_consensus.cc:504] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:32.038318 13301 leader_election.cc:288] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:32.038717 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:32.038735 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:32.038870 11381 raft_consensus.cc:2454] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:32.038870 11249 raft_consensus.cc:2454] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:32.039157 11447 leader_election.cc:302] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:32.039333 13301 raft_consensus.cc:2781] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:32.039400 13301 raft_consensus.cc:482] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:32.039429 13301 raft_consensus.cc:3037] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.040287 13301 raft_consensus.cc:504] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:32.040602 13301 leader_election.cc:288] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:32.040897 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:32.040921 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:32.041015 11249 raft_consensus.cc:3037] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.041049 11381 raft_consensus.cc:3037] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.041795 11381 raft_consensus.cc:2454] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:32.041831 11249 raft_consensus.cc:2454] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:32.042068 11447 leader_election.cc:302] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:32.042271 13301 raft_consensus.cc:2781] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:32.042344 13301 raft_consensus.cc:686] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:32.042479 13301 consensus_queue.cc:237] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:32.045356 11098 catalog_manager.cc:5484] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:32.058315 13301 raft_consensus.cc:482] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:32.058445 13301 raft_consensus.cc:504] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.058828 13301 leader_election.cc:288] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:32.059208 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe34db1ffe5f4509abb547f88c12a562" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:32.059208 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe34db1ffe5f4509abb547f88c12a562" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:32.059340 11643 raft_consensus.cc:2454] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:32.059347 11381 raft_consensus.cc:2454] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:32.059638 11447 leader_election.cc:302] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:32.059804 13301 raft_consensus.cc:2781] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:32.059849 13301 raft_consensus.cc:482] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:32.059873 13301 raft_consensus.cc:3037] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.060714 13301 raft_consensus.cc:504] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.061050 13301 leader_election.cc:288] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:32.061371 11381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe34db1ffe5f4509abb547f88c12a562" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:32.061424 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe34db1ffe5f4509abb547f88c12a562" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:32.061523 11381 raft_consensus.cc:3037] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.061561 11643 raft_consensus.cc:3037] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.062521 11643 raft_consensus.cc:2454] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:32.062518 11381 raft_consensus.cc:2454] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:32.062793 11446 leader_election.cc:302] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:32.063017 13301 raft_consensus.cc:2781] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:32.063122 13301 raft_consensus.cc:686] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:32.063274 13301 consensus_queue.cc:237] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.065580 11098 catalog_manager.cc:5484] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:32.155829 13220 raft_consensus.cc:482] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:32.155982 13220 raft_consensus.cc:504] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.156414 13220 leader_election.cc:288] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:32.156728 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b57520d53734c318d603dd46e30b190" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:32.156824 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b57520d53734c318d603dd46e30b190" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:32.156916 11511 raft_consensus.cc:2454] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:32.156988 11643 raft_consensus.cc:2454] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:32.157228 11181 leader_election.cc:302] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:32.157397 13220 raft_consensus.cc:2781] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:32.157459 13220 raft_consensus.cc:482] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:32.157493 13220 raft_consensus.cc:3037] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.158314 13220 raft_consensus.cc:504] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.158638 13220 leader_election.cc:288] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:32.158979 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b57520d53734c318d603dd46e30b190" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:32.158982 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b57520d53734c318d603dd46e30b190" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:32.159088 11643 raft_consensus.cc:3037] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.159090 11511 raft_consensus.cc:3037] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:32.159837 11643 raft_consensus.cc:2454] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:32.159989 11511 raft_consensus.cc:2454] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:32.160132 11182 leader_election.cc:302] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:32.160305 13220 raft_consensus.cc:2781] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:32.160409 13220 raft_consensus.cc:686] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:32.160545 13220 consensus_queue.cc:237] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:32.162515 11098 catalog_manager.cc:5484] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:32.301941 13550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:32.302202 13550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:32.305545 13550 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 19:56:32.396068 11249 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:32.396112 11381 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:32.397459 13301 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:32.398881 13301 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:32.400853 11643 raft_consensus.cc:1261] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:32.401242 13434 consensus_queue.cc:1035] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:32.402608 11511 raft_consensus.cc:1261] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:32.403667 13220 consensus_queue.cc:1035] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:32.409647 11378 raft_consensus.cc:1261] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:32.409691 11643 raft_consensus.cc:1261] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:32.410077 13155 consensus_queue.cc:1035] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:32.410373 13155 consensus_queue.cc:1035] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.357 ms
time per row: 0.0596787 ms
I20240501 19:56:32.471172 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.5:42501
--local_ip_for_outbound_sockets=127.5.40.5
--tserver_master_addrs=127.5.40.62:42037
--webserver_port=36289
--webserver_interface=127.5.40.5
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:32.592254 13588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:32.592486 13588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:32.592543 13588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:32.595762 13588 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:32.595839 13588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:32.595913 13588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.5
I20240501 19:56:32.599408 13588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.5:42501
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.40.5
--webserver_port=36289
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.13588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.5
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:32.600271 13588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:32.601466 13588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:32.609874 13593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:32.610147 13596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:32.610590 13594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:32.611019 13588 server_base.cc:1034] running on GCE node
I20240501 19:56:32.611429 13588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:32.612120 13588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:32.613281 13588 hybrid_clock.cc:648] HybridClock initialized: now 1714593392613254 us; error 41 us; skew 500 ppm
I20240501 19:56:32.615530 13588 webserver.cc:460] Webserver started at http://127.5.40.5:36289/ using document root <none> and password file <none>
I20240501 19:56:32.616134 13588 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:32.616210 13588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:32.619606 13588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 19:56:32.621552 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.622575 13588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:32.622684 13588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-bz40"
I20240501 19:56:32.623098 13588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:32.641458 13588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:32.642117 13588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:32.642261 13588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:32.642781 13588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:32.643712 13588 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:32.643764 13588 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.643818 13588 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:32.643841 13588 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.652452 13588 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.5:42501
I20240501 19:56:32.652766 13715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.5:42501 every 8 connection(s)
I20240501 19:56:32.653681 13588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:32.656888 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 13588
I20240501 19:56:32.657256 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.6:46331
--local_ip_for_outbound_sockets=127.5.40.6
--tserver_master_addrs=127.5.40.62:42037
--webserver_port=45411
--webserver_interface=127.5.40.6
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:32.664486 13716 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:32.664765 13716 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:32.665501 13716 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:32.666759 11105 ts_manager.cc:194] Re-registered known tserver with Master: 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:32.667382 11105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.5:54459
W20240501 19:56:32.772680 13719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:32.772953 13719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:32.773003 13719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:32.776546 13719 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:32.776635 13719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:32.776710 13719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.6
I20240501 19:56:32.780678 13719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.6:46331
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.40.6
--webserver_port=45411
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.13719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.6
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:32.781850 13719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:32.783015 13719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:32.791684 13727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:32.792016 13725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:32.792384 13724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:32.793015 13719 server_base.cc:1034] running on GCE node
I20240501 19:56:32.793424 13719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:32.794044 13719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:32.795222 13719 hybrid_clock.cc:648] HybridClock initialized: now 1714593392795198 us; error 33 us; skew 500 ppm
I20240501 19:56:32.797381 13719 webserver.cc:460] Webserver started at http://127.5.40.6:45411/ using document root <none> and password file <none>
I20240501 19:56:32.797972 13719 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:32.798046 13719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:32.802073 13719 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:32.803997 13733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.805075 13719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:32.805178 13719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:32.805617 13719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:32.823385 13719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:32.824059 13719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:32.824218 13719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:32.824761 13719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:32.825676 13719 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:32.825722 13719 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.825775 13719 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:32.825803 13719 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.834399 13719 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.6:46331
I20240501 19:56:32.834522 13846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.6:46331 every 8 connection(s)
I20240501 19:56:32.835510 13719 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:32.843011 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 13719
I20240501 19:56:32.843405 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.7:36243
--local_ip_for_outbound_sockets=127.5.40.7
--tserver_master_addrs=127.5.40.62:42037
--webserver_port=35033
--webserver_interface=127.5.40.7
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:32.844699 13847 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:32.844951 13847 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:32.845571 13847 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:32.846694 11105 ts_manager.cc:194] Re-registered known tserver with Master: 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:32.847394 11105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.6:45777
W20240501 19:56:32.961091 13850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:32.961371 13850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:32.961448 13850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:32.965066 13850 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:32.965142 13850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:32.965230 13850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.7
I20240501 19:56:32.969226 13850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.7:36243
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.40.7
--webserver_port=35033
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.13850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.7
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:32.970268 13850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:32.971403 13850 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:32.979580 13858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:32.979923 13855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:32.980394 13856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:32.980939 13850 server_base.cc:1034] running on GCE node
I20240501 19:56:32.981393 13850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:32.982092 13850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:32.983258 13850 hybrid_clock.cc:648] HybridClock initialized: now 1714593392983238 us; error 36 us; skew 500 ppm
I20240501 19:56:32.985204 13850 webserver.cc:460] Webserver started at http://127.5.40.7:35033/ using document root <none> and password file <none>
I20240501 19:56:32.985721 13850 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:32.985783 13850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:32.990075 13850 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 19:56:32.991984 13864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.992974 13850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:32.993070 13850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6921cf2e41f64d66842885512da8de6c"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:32.993463 13850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:33.008117 13850 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:33.008754 13850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:33.008910 13850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:33.009438 13850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:33.010380 13850 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:33.010427 13850 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:33.010488 13850 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:33.010512 13850 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:33.019271 13850 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.7:36243
I20240501 19:56:33.019315 13977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.7:36243 every 8 connection(s)
I20240501 19:56:33.020563 13850 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:33.030427 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 13850
I20240501 19:56:33.030896 5280 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
/tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.40.8:34789
--local_ip_for_outbound_sockets=127.5.40.8
--tserver_master_addrs=127.5.40.62:42037
--webserver_port=43041
--webserver_interface=127.5.40.8
--builtin_ntp_servers=127.5.40.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:33.033901 13978 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:33.034155 13978 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:33.034812 13978 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:33.036033 11105 ts_manager.cc:194] Re-registered known tserver with Master: 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:33.036801 11105 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.7:56779
W20240501 19:56:33.155968 13981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:33.156282 13981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:33.156395 13981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:33.159888 13981 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:33.159955 13981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:33.160025 13981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.40.8
I20240501 19:56:33.163983 13981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.40.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.40.8:34789
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.40.8
--webserver_port=43041
--tserver_master_addrs=127.5.40.62:42037
--never_fsync=true
--heap_profile_path=/tmp/kudu.13981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.40.8
--log_dir=/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:33.165118 13981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:33.166272 13981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:33.175189 13987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:33.175664 13989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:33.175983 13986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:33.176497 13981 server_base.cc:1034] running on GCE node
I20240501 19:56:33.176890 13981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:33.177475 13981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:33.178613 13981 hybrid_clock.cc:648] HybridClock initialized: now 1714593393178597 us; error 33 us; skew 500 ppm
I20240501 19:56:33.180511 13981 webserver.cc:460] Webserver started at http://127.5.40.8:43041/ using document root <none> and password file <none>
I20240501 19:56:33.181061 13981 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:33.181128 13981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:33.184830 13981 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 19:56:33.187270 13995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:33.188357 13981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:33.188464 13981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "533908e517b24f77b2af8b5863e4ae5a"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-bz40"
I20240501 19:56:33.188938 13981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:33.205179 13981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:33.205827 13981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:33.205978 13981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:33.206470 13981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:33.207330 13981 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:33.207376 13981 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:33.207415 13981 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:33.207458 13981 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:33.215467 13981 rpc_server.cc:307] RPC server started. Bound to: 127.5.40.8:34789
I20240501 19:56:33.215500 14108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.40.8:34789 every 8 connection(s)
I20240501 19:56:33.216557 13981 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:33.217818 5280 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktBk3Nr/build/debug/bin/kudu as pid 13981
I20240501 19:56:33.221707 11105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:33.229422 11358 tablet_service.cc:1467] Processing CreateTablet for tablet f992221f642b41c3861ed4efb1f48e6c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cc96dbcfc5a4808a4bcc0842245d91a]), partition=
I20240501 19:56:33.229702 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f992221f642b41c3861ed4efb1f48e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.229959 14109 heartbeater.cc:347] Connected to a master server at 127.5.40.62:42037
I20240501 19:56:33.230194 14109 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:33.230180 11491 tablet_service.cc:1467] Processing CreateTablet for tablet f992221f642b41c3861ed4efb1f48e6c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cc96dbcfc5a4808a4bcc0842245d91a]), partition=
I20240501 19:56:33.230445 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f992221f642b41c3861ed4efb1f48e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.230839 14109 heartbeater.cc:511] Master 127.5.40.62:42037 requested a full tablet report, sending...
I20240501 19:56:33.231844 11099 ts_manager.cc:194] Re-registered known tserver with Master: 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:33.232540 11099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.40.8:46351
I20240501 19:56:33.233183 14123 tablet_bootstrap.cc:492] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:33.233790 14124 tablet_bootstrap.cc:492] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:33.234568 14124 tablet_bootstrap.cc:654] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.235134 14123 tablet_bootstrap.cc:654] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.236124 14123 tablet_bootstrap.cc:492] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:33.236215 14123 ts_tablet_manager.cc:1367] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:33.236605 14123 raft_consensus.cc:348] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:33.236714 14123 raft_consensus.cc:374] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.236744 14123 raft_consensus.cc:729] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.236862 14123 consensus_queue.cc:260] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:33.237155 14123 ts_tablet_manager.cc:1398] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.238679 14124 tablet_bootstrap.cc:492] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:33.238819 14124 ts_tablet_manager.cc:1367] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:33.239476 14124 raft_consensus.cc:348] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:33.239665 14124 raft_consensus.cc:374] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.239742 14124 raft_consensus.cc:729] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.239938 14124 consensus_queue.cc:260] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:33.242699 14124 ts_tablet_manager.cc:1398] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:33.243594 13912 tablet_service.cc:1467] Processing CreateTablet for tablet f992221f642b41c3861ed4efb1f48e6c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cc96dbcfc5a4808a4bcc0842245d91a]), partition=
I20240501 19:56:33.244359 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f992221f642b41c3861ed4efb1f48e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.250377 14127 tablet_bootstrap.cc:492] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:33.252956 14127 tablet_bootstrap.cc:654] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.254307 14127 log.cc:824] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:33.255990 14127 tablet_bootstrap.cc:492] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:33.256161 14127 ts_tablet_manager.cc:1367] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 19:56:33.259145 14127 raft_consensus.cc:348] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:33.259320 14127 raft_consensus.cc:374] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.259353 14127 raft_consensus.cc:729] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.259783 14127 consensus_queue.cc:260] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:33.260440 13978 heartbeater.cc:503] Master 127.5.40.62:42037 was elected leader, sending a full tablet report...
I20240501 19:56:33.260576 14127 ts_tablet_manager.cc:1398] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20240501 19:56:33.271679 13979 tablet.cc:2369] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:33.391881 13209 raft_consensus.cc:482] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.392020 13209 raft_consensus.cc:504] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:33.392488 13209 leader_election.cc:288] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:33.393540 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f992221f642b41c3861ed4efb1f48e6c" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:33.393707 11508 raft_consensus.cc:2454] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:33.394366 11313 leader_election.cc:302] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:33.394637 13209 raft_consensus.cc:2781] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.394703 13209 raft_consensus.cc:482] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.394738 13209 raft_consensus.cc:3037] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.395793 13209 raft_consensus.cc:504] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:33.396167 13209 leader_election.cc:288] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:33.396682 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f992221f642b41c3861ed4efb1f48e6c" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:33.396803 11508 raft_consensus.cc:3037] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.397413 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f992221f642b41c3861ed4efb1f48e6c" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:33.397413 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f992221f642b41c3861ed4efb1f48e6c" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:33.397729 13932 raft_consensus.cc:2454] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:33.397768 11508 raft_consensus.cc:2454] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:33.398054 11313 leader_election.cc:302] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:33.398231 13209 raft_consensus.cc:2781] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.398388 13209 raft_consensus.cc:686] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:33.398533 13209 consensus_queue.cc:237] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:33.400467 11105 catalog_manager.cc:5484] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:33.487758 11624 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:33.494361 11358 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:33.498355 13912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:33.499017 14043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:33.500964 11227 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:33.501415 13781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:33.513423 11491 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:56:33.529729 11099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:33.537720 13932 raft_consensus.cc:3037] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.538985 13932 raft_consensus.cc:1261] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.539462 11508 raft_consensus.cc:1261] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:33.544086 13209 consensus_queue.cc:1035] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.544330 14151 consensus_queue.cc:1035] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:33.547989 11099 catalog_manager.cc:5576] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965) does not have the latest schema for tablet f992221f642b41c3861ed4efb1f48e6c (table rebalancer_dynamic_table [id=9cc96dbcfc5a4808a4bcc0842245d91a]). Expected version 1 got 0
I20240501 19:56:33.550482 11102 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet f992221f642b41c3861ed4efb1f48e6c (table rebalancer_dynamic_table [id=9cc96dbcfc5a4808a4bcc0842245d91a]). Expected version 1 got 0
I20240501 19:56:33.551759 13650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:33.553820 11102 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet f992221f642b41c3861ed4efb1f48e6c (table rebalancer_dynamic_table [id=9cc96dbcfc5a4808a4bcc0842245d91a]). Expected version 1 got 0
I20240501 19:56:33.556583 14195 mvcc.cc:205] Tried to move back new op lower bound from 7022974539921612800 to 7022974539361779712. Current Snapshot: MvccSnapshot[applied={T|T < 7022974539921612800}]
I20240501 19:56:33.557495 14198 tablet.cc:1722] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:33.558457 14198 tablet.cc:1620] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:33.558478 14197 tablet.cc:1722] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:33.559162 14197 tablet.cc:1620] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: MemRowSet was empty: no flush needed.
I20240501 19:56:33.565341 11099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=9cc96dbcfc5a4808a4bcc0842245d91a] alter complete (version 1)
I20240501 19:56:33.566429 11099 catalog_manager.cc:5576] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965) does not have the latest schema for tablet f992221f642b41c3861ed4efb1f48e6c (table rebalancer_dynamic_table [id=9cc96dbcfc5a4808a4bcc0842245d91a]). Expected version 1 got 0
I20240501 19:56:33.568414 14200 tablet.cc:1722] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:33.569411 14200 tablet.cc:1620] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:33.669862 13716 heartbeater.cc:503] Master 127.5.40.62:42037 was elected leader, sending a full tablet report...
I20240501 19:56:33.669994 11098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:33.677210 14197 tablet.cc:1722] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:33.677572 14197 tablet.cc:1620] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: MemRowSet was empty: no flush needed.
I20240501 19:56:33.681264 14200 tablet.cc:1722] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:33.681596 14200 tablet.cc:1620] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:33.686722 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=9cc96dbcfc5a4808a4bcc0842245d91a] alter complete (version 2)
I20240501 19:56:33.688918 14198 tablet.cc:1722] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:33.689230 14198 tablet.cc:1620] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:33.698930 11624 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:33.701550 11227 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:33.703002 13650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:33.704422 13781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:33.712363 14043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:33.712584 13912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:33.719113 11098 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 19:56:33.719300 11098 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 19:56:33.722394 11098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 171bb2e007d1427ebf737bb9fa8b75a8
I20240501 19:56:33.722859 11098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet b3a53d7da1204a2a9cb0869330285284
I20240501 19:56:33.723017 11491 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:56:33.723102 11098 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 9cd620de3ef44eab9cd021874be0a3e4
I20240501 19:56:33.723273 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 171bb2e007d1427ebf737bb9fa8b75a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:33.723353 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a53d7da1204a2a9cb0869330285284 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:33.723901 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 171bb2e007d1427ebf737bb9fa8b75a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:33.723940 11490 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cd620de3ef44eab9cd021874be0a3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:33.724210 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a53d7da1204a2a9cb0869330285284 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:33.724330 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a53d7da1204a2a9cb0869330285284 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:33.724552 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cd620de3ef44eab9cd021874be0a3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:33.724210 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet 171bb2e007d1427ebf737bb9fa8b75a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:33.724798 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cd620de3ef44eab9cd021874be0a3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:33.727887 14260 tablet_replica.cc:330] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:33.728195 14260 raft_consensus.cc:2229] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:33.728370 14260 raft_consensus.cc:2258] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.729287 14260 ts_tablet_manager.cc:1875] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.732687 14258 tablet_replica.cc:330] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:33.732703 14260 ts_tablet_manager.cc:1888] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:33.732779 14260 log.cc:1196] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/9cd620de3ef44eab9cd021874be0a3e4
I20240501 19:56:33.733345 14258 raft_consensus.cc:2229] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:33.733500 14260 ts_tablet_manager.cc:1909] T 9cd620de3ef44eab9cd021874be0a3e4 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:33.733680 14258 raft_consensus.cc:2258] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.733963 11104 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:33.734665 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 9cd620de3ef44eab9cd021874be0a3e4 (table rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]) successfully deleted
I20240501 19:56:33.734985 14258 ts_tablet_manager.cc:1875] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.735422 14260 tablet_replica.cc:330] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:33.735589 14260 raft_consensus.cc:2229] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:33.735913 14260 raft_consensus.cc:2258] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.736636 14260 ts_tablet_manager.cc:1875] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.737085 11358 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:33.737200 14261 tablet_replica.cc:330] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:33.737510 14261 raft_consensus.cc:2229] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:33.737691 14261 raft_consensus.cc:2258] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.738392 14258 ts_tablet_manager.cc:1888] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:33.738472 14258 log.cc:1196] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/171bb2e007d1427ebf737bb9fa8b75a8
I20240501 19:56:33.738723 14258 ts_tablet_manager.cc:1909] T 171bb2e007d1427ebf737bb9fa8b75a8 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:33.739948 14261 ts_tablet_manager.cc:1875] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.741039 14260 ts_tablet_manager.cc:1888] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:33.741111 14260 log.cc:1196] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/b3a53d7da1204a2a9cb0869330285284
I20240501 19:56:33.741334 14260 ts_tablet_manager.cc:1909] T b3a53d7da1204a2a9cb0869330285284 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:33.742015 14261 ts_tablet_manager.cc:1888] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:33.742092 14261 log.cc:1196] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/b3a53d7da1204a2a9cb0869330285284
I20240501 19:56:33.742246 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet b3a53d7da1204a2a9cb0869330285284 (table rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]) successfully deleted
I20240501 19:56:33.742370 14261 ts_tablet_manager.cc:1909] T b3a53d7da1204a2a9cb0869330285284 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:33.743160 14261 tablet_replica.cc:330] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:33.743268 14261 raft_consensus.cc:2229] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:33.743242 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet b3a53d7da1204a2a9cb0869330285284 (table rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]) successfully deleted
I20240501 19:56:33.743491 14261 raft_consensus.cc:2258] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.743888 14261 ts_tablet_manager.cc:1875] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.745285 14258 tablet_replica.cc:330] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:33.745358 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 171bb2e007d1427ebf737bb9fa8b75a8 (table rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]) successfully deleted
I20240501 19:56:33.745535 14258 raft_consensus.cc:2229] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:33.745836 14258 raft_consensus.cc:2258] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.746364 14258 ts_tablet_manager.cc:1875] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.747924 14258 ts_tablet_manager.cc:1888] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:33.747987 14258 log.cc:1196] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/9cd620de3ef44eab9cd021874be0a3e4
I20240501 19:56:33.748201 14258 ts_tablet_manager.cc:1909] T 9cd620de3ef44eab9cd021874be0a3e4 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:33.748775 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 9cd620de3ef44eab9cd021874be0a3e4 (table rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]) successfully deleted
I20240501 19:56:33.749174 14261 ts_tablet_manager.cc:1888] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:33.749235 14261 log.cc:1196] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/171bb2e007d1427ebf737bb9fa8b75a8
I20240501 19:56:33.749437 14261 ts_tablet_manager.cc:1909] T 171bb2e007d1427ebf737bb9fa8b75a8 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:33.750065 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 171bb2e007d1427ebf737bb9fa8b75a8 (table rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]) successfully deleted
I20240501 19:56:33.755676 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2498588acd4be0a23017da1b713182 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.755993 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2498588acd4be0a23017da1b713182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.758266 11358 tablet_service.cc:1467] Processing CreateTablet for tablet d7bb9ef27c6e4f248a0e5a4812986f38 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.758558 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7bb9ef27c6e4f248a0e5a4812986f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.759168 13911 tablet_service.cc:1467] Processing CreateTablet for tablet d7bb9ef27c6e4f248a0e5a4812986f38 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.759563 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7bb9ef27c6e4f248a0e5a4812986f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.761034 11227 tablet_service.cc:1467] Processing CreateTablet for tablet d7bb9ef27c6e4f248a0e5a4812986f38 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.761312 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7bb9ef27c6e4f248a0e5a4812986f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.759161 13912 tablet_service.cc:1467] Processing CreateTablet for tablet ba2b4dda83bc408eaddec5ebc7b734d5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.763223 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b4dda83bc408eaddec5ebc7b734d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.763597 14127 tablet_bootstrap.cc:492] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:33.764437 14127 tablet_bootstrap.cc:654] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.764518 14265 tablet_bootstrap.cc:492] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:33.765357 14265 tablet_bootstrap.cc:654] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.771152 14265 tablet_bootstrap.cc:492] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:33.771627 14265 ts_tablet_manager.cc:1367] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:33.772328 14265 raft_consensus.cc:348] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.772565 14265 raft_consensus.cc:374] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.772622 14265 raft_consensus.cc:729] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.772769 14265 consensus_queue.cc:260] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.773094 14265 ts_tablet_manager.cc:1398] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.773823 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2498588acd4be0a23017da1b713182 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.774878 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2498588acd4be0a23017da1b713182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.777258 14259 tablet_replica.cc:330] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:33.777261 14043 tablet_service.cc:1467] Processing CreateTablet for tablet ba2b4dda83bc408eaddec5ebc7b734d5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.777822 14259 raft_consensus.cc:2229] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:33.778147 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b4dda83bc408eaddec5ebc7b734d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.778148 14259 raft_consensus.cc:2258] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.779142 14259 ts_tablet_manager.cc:1875] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.781587 14259 ts_tablet_manager.cc:1888] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:33.781737 14259 log.cc:1196] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/b3a53d7da1204a2a9cb0869330285284
I20240501 19:56:33.782101 14259 ts_tablet_manager.cc:1909] T b3a53d7da1204a2a9cb0869330285284 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:33.783553 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet b3a53d7da1204a2a9cb0869330285284 (table rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]) successfully deleted
I20240501 19:56:33.783802 14259 tablet_replica.cc:330] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:33.783991 14259 raft_consensus.cc:2229] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:33.784150 14259 raft_consensus.cc:2258] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.784135 13781 tablet_service.cc:1467] Processing CreateTablet for tablet ba2b4dda83bc408eaddec5ebc7b734d5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.784735 14259 ts_tablet_manager.cc:1875] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.785071 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b4dda83bc408eaddec5ebc7b734d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.785423 14270 tablet_bootstrap.cc:492] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:33.786876 14259 ts_tablet_manager.cc:1888] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:33.786935 14259 log.cc:1196] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/171bb2e007d1427ebf737bb9fa8b75a8
I20240501 19:56:33.787153 14259 ts_tablet_manager.cc:1909] T 171bb2e007d1427ebf737bb9fa8b75a8 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:33.787479 14270 tablet_bootstrap.cc:654] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.787528 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2498588acd4be0a23017da1b713182 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:33.787763 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2498588acd4be0a23017da1b713182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.787788 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 171bb2e007d1427ebf737bb9fa8b75a8 (table rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]) successfully deleted
I20240501 19:56:33.787710 14259 tablet_replica.cc:330] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:33.788054 14259 raft_consensus.cc:2229] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:33.788174 14259 raft_consensus.cc:2258] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.788290 14270 log.cc:824] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:33.788785 11098 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:33.788954 14259 ts_tablet_manager.cc:1875] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.790671 14259 ts_tablet_manager.cc:1888] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:33.790727 14259 log.cc:1196] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/9cd620de3ef44eab9cd021874be0a3e4
I20240501 19:56:33.791244 14259 ts_tablet_manager.cc:1909] T 9cd620de3ef44eab9cd021874be0a3e4 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:33.791338 14267 tablet_bootstrap.cc:492] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:33.792135 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 9cd620de3ef44eab9cd021874be0a3e4 (table rebalance_test_table_0 [id=f45f3dce230248b283052c9719393f58]) successfully deleted
I20240501 19:56:33.792742 14267 tablet_bootstrap.cc:654] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.792881 14127 tablet_bootstrap.cc:492] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:33.792965 14127 ts_tablet_manager.cc:1367] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 19:56:33.793537 14127 raft_consensus.cc:348] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:33.793654 14127 raft_consensus.cc:374] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.793682 14127 raft_consensus.cc:729] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.793797 14127 consensus_queue.cc:260] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:33.794054 14127 ts_tablet_manager.cc:1398] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.794148 14266 tablet_bootstrap.cc:492] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:33.794217 14127 tablet_bootstrap.cc:492] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:33.795042 14127 tablet_bootstrap.cc:654] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.799885 14266 tablet_bootstrap.cc:654] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.801165 14272 tablet_bootstrap.cc:492] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:33.803392 14267 tablet_bootstrap.cc:492] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:33.803516 14272 tablet_bootstrap.cc:654] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.803603 14267 ts_tablet_manager.cc:1367] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240501 19:56:33.804064 14267 raft_consensus.cc:348] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:33.804242 14267 raft_consensus.cc:374] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.804318 14267 raft_consensus.cc:729] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.804566 14197 tablet.cc:1722] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:33.804721 14272 log.cc:824] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:33.804896 14267 consensus_queue.cc:260] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:33.805317 14267 ts_tablet_manager.cc:1398] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:33.805553 14198 tablet.cc:1722] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:33.810478 14200 tablet.cc:1722] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:33.810771 11098 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=9cc96dbcfc5a4808a4bcc0842245d91a] alter complete (version 3)
I20240501 19:56:33.813163 14127 tablet_bootstrap.cc:492] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:33.813261 14127 ts_tablet_manager.cc:1367] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20240501 19:56:33.813737 14127 raft_consensus.cc:348] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.813855 14127 raft_consensus.cc:374] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.813890 14127 raft_consensus.cc:729] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.814010 14127 consensus_queue.cc:260] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.814252 14127 ts_tablet_manager.cc:1398] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:33.816787 14270 tablet_bootstrap.cc:492] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:33.817018 14270 ts_tablet_manager.cc:1367] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240501 19:56:33.820767 14270 raft_consensus.cc:348] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.820981 14270 raft_consensus.cc:374] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.821030 14270 raft_consensus.cc:729] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.821427 14270 consensus_queue.cc:260] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.822134 14109 heartbeater.cc:503] Master 127.5.40.62:42037 was elected leader, sending a full tablet report...
I20240501 19:56:33.822373 14270 ts_tablet_manager.cc:1398] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:33.822427 14274 tablet_bootstrap.cc:492] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:33.823173 14266 tablet_bootstrap.cc:492] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:33.823277 14266 ts_tablet_manager.cc:1367] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 19:56:33.823725 14266 raft_consensus.cc:348] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:33.823828 14266 raft_consensus.cc:374] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.823854 14266 raft_consensus.cc:729] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.823969 14266 consensus_queue.cc:260] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:33.824244 14266 ts_tablet_manager.cc:1398] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.824357 14272 tablet_bootstrap.cc:492] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:33.824539 14272 ts_tablet_manager.cc:1367] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20240501 19:56:33.824802 14274 tablet_bootstrap.cc:654] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.825771 14274 log.cc:824] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:33.827932 14272 raft_consensus.cc:348] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.828152 14272 raft_consensus.cc:374] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.828209 14272 raft_consensus.cc:729] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.828682 14272 consensus_queue.cc:260] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.829546 14274 tablet_bootstrap.cc:492] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:33.829713 14274 ts_tablet_manager.cc:1367] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 19:56:33.829716 14272 ts_tablet_manager.cc:1398] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:33.833302 14274 raft_consensus.cc:348] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.833528 14274 raft_consensus.cc:374] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.833623 14274 raft_consensus.cc:729] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.834095 14274 consensus_queue.cc:260] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.834789 13847 heartbeater.cc:503] Master 127.5.40.62:42037 was elected leader, sending a full tablet report...
I20240501 19:56:33.836120 14274 ts_tablet_manager.cc:1398] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:56:33.836452 14274 tablet_bootstrap.cc:492] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
W20240501 19:56:33.837358 13848 tablet.cc:2369] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:33.838582 14274 tablet_bootstrap.cc:654] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:33.839192 14191 raft_consensus.cc:482] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.839283 14191 raft_consensus.cc:504] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.839766 14191 leader_election.cc:288] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:33.847470 13670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2498588acd4be0a23017da1b713182" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:33.847574 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2498588acd4be0a23017da1b713182" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:33.847795 13670 raft_consensus.cc:2454] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:33.847795 13801 raft_consensus.cc:2454] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:33.848177 14274 tablet_bootstrap.cc:492] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:33.848150 11580 leader_election.cc:302] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:33.848279 14274 ts_tablet_manager.cc:1367] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:33.848881 14274 raft_consensus.cc:348] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.848989 14274 raft_consensus.cc:374] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:33.849020 14274 raft_consensus.cc:729] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:33.849128 14274 consensus_queue.cc:260] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.849349 14274 ts_tablet_manager.cc:1398] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:33.849493 14191 raft_consensus.cc:2781] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.849536 14191 raft_consensus.cc:482] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.849560 14191 raft_consensus.cc:3037] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.850355 14191 raft_consensus.cc:504] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.850733 14191 leader_election.cc:288] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:33.851222 13670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2498588acd4be0a23017da1b713182" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:33.851294 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2498588acd4be0a23017da1b713182" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:33.851351 13670 raft_consensus.cc:3037] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.851382 13801 raft_consensus.cc:3037] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.852416 13801 raft_consensus.cc:2454] T 6f2498588acd4be0a23017da1b713182 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:33.852416 13670 raft_consensus.cc:2454] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:33.852684 11580 leader_election.cc:302] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:33.852910 14191 raft_consensus.cc:2781] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.853034 14191 raft_consensus.cc:686] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:33.853168 14191 consensus_queue.cc:237] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.855325 11106 catalog_manager.cc:5484] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:33.906177 13717 tablet.cc:2369] T 6f2498588acd4be0a23017da1b713182 P 8125d8ad6d0d44caae9c5c90c90df96b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:33.913528 11101 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:33.913684 11101 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:33.915868 11101 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet f992221f642b41c3861ed4efb1f48e6c
I20240501 19:56:33.916369 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet f992221f642b41c3861ed4efb1f48e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:33.916545 14261 tablet_replica.cc:330] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:33.916661 14261 raft_consensus.cc:2229] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:33.916864 14261 raft_consensus.cc:2258] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.917093 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet f992221f642b41c3861ed4efb1f48e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:33.917886 14261 ts_tablet_manager.cc:1875] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.918622 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet f992221f642b41c3861ed4efb1f48e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:33 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:33.918869 14260 tablet_replica.cc:330] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:33.919008 14260 raft_consensus.cc:2229] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:33.919121 14260 raft_consensus.cc:2258] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.920145 14261 ts_tablet_manager.cc:1888] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:33.920219 14261 log.cc:1196] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/f992221f642b41c3861ed4efb1f48e6c
I20240501 19:56:33.920481 14261 ts_tablet_manager.cc:1909] T f992221f642b41c3861ed4efb1f48e6c P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:33.921233 14129 raft_consensus.cc:482] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.921331 14129 raft_consensus.cc:504] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:33.921424 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet f992221f642b41c3861ed4efb1f48e6c (table rebalancer_dynamic_table_new_name [id=9cc96dbcfc5a4808a4bcc0842245d91a]) successfully deleted
I20240501 19:56:33.922269 14260 ts_tablet_manager.cc:1875] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.922662 14129 leader_election.cc:288] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:33.924507 14260 ts_tablet_manager.cc:1888] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:33.924572 14260 log.cc:1196] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/f992221f642b41c3861ed4efb1f48e6c
I20240501 19:56:33.926734 14260 ts_tablet_manager.cc:1909] T f992221f642b41c3861ed4efb1f48e6c P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:33.928617 14327 tablet_replica.cc:330] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:33.928961 14327 raft_consensus.cc:2229] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:33.929188 14327 raft_consensus.cc:2258] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:33.930490 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bb9ef27c6e4f248a0e5a4812986f38" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:33.930634 11248 raft_consensus.cc:2454] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:33.930984 13865 leader_election.cc:302] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:33.931217 14129 raft_consensus.cc:2781] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.931267 14129 raft_consensus.cc:482] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.931301 14129 raft_consensus.cc:3037] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.931676 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet f992221f642b41c3861ed4efb1f48e6c (table rebalancer_dynamic_table_new_name [id=9cc96dbcfc5a4808a4bcc0842245d91a]) successfully deleted
I20240501 19:56:33.932346 14129 raft_consensus.cc:504] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:33.932622 14327 ts_tablet_manager.cc:1875] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:33.932719 14129 leader_election.cc:288] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:33.933352 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bb9ef27c6e4f248a0e5a4812986f38" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:33.933466 11248 raft_consensus.cc:3037] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.934425 11248 raft_consensus.cc:2454] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:33.934801 13865 leader_election.cc:302] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:33.935050 14129 raft_consensus.cc:2781] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.935220 14129 raft_consensus.cc:686] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Becoming Leader. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Running, Role: LEADER
I20240501 19:56:33.935503 14129 consensus_queue.cc:237] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:33.938622 11101 catalog_manager.cc:5484] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921cf2e41f64d66842885512da8de6c (127.5.40.7). New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:33.938815 14327 ts_tablet_manager.cc:1888] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:33.938899 14327 log.cc:1196] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/f992221f642b41c3861ed4efb1f48e6c
I20240501 19:56:33.939237 14327 ts_tablet_manager.cc:1909] T f992221f642b41c3861ed4efb1f48e6c P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:33.941332 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bb9ef27c6e4f248a0e5a4812986f38" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:33.941385 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bb9ef27c6e4f248a0e5a4812986f38" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:33.941514 11377 raft_consensus.cc:2454] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:33.942605 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet f992221f642b41c3861ed4efb1f48e6c (table rebalancer_dynamic_table_new_name [id=9cc96dbcfc5a4808a4bcc0842245d91a]) successfully deleted
I20240501 19:56:33.945458 14330 raft_consensus.cc:482] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.945556 14330 raft_consensus.cc:504] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.949273 14330 leader_election.cc:288] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:33.952529 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:33.952822 14063 raft_consensus.cc:2454] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:33.953366 13865 leader_election.cc:302] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:33.954082 14330 raft_consensus.cc:2781] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:33.954193 14330 raft_consensus.cc:482] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:33.954435 14330 raft_consensus.cc:3037] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.956380 14330 raft_consensus.cc:504] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.957070 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:33.957194 14063 raft_consensus.cc:3037] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.958348 14063 raft_consensus.cc:2454] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:33.958815 14281 raft_consensus.cc:482] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:33.958931 13865 leader_election.cc:302] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:33.958918 14281 raft_consensus.cc:504] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.959340 14129 raft_consensus.cc:2781] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:33.959606 14129 raft_consensus.cc:686] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Becoming Leader. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Running, Role: LEADER
I20240501 19:56:33.960016 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:33.960125 13801 raft_consensus.cc:2454] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:33.960199 14281 leader_election.cc:288] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:33.960569 14129 consensus_queue.cc:237] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:33.962832 11104 catalog_manager.cc:5484] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921cf2e41f64d66842885512da8de6c (127.5.40.7). New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:33.967481 14330 leader_election.cc:288] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:33.967813 14110 tablet.cc:2369] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:33.967854 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:33.967994 13801 raft_consensus.cc:3037] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:33.968991 13801 raft_consensus.cc:2454] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:33.975659 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:33.977316 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:33.977530 14063 raft_consensus.cc:2379] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in current term 1: Already voted for candidate 6921cf2e41f64d66842885512da8de6c in this term.
I20240501 19:56:33.978515 13734 leader_election.cc:302] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96fc18dbb5fd41afb9bebf805e8d33a7; no voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:33.978974 14281 raft_consensus.cc:2726] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:33.989041 14043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:33.992738 11227 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:33.998379 13912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:33.999372 11358 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:34.000242 11624 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 19:56:34.002441 11491 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 19:56:34.008764 13650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:34.022724 11102 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:34.028982 13781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:34.034175 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 86a125b33b0a4c109a27d7626ec4568e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1407a4f230fa45469e0f6168d7549f59]), partition=
I20240501 19:56:34.034452 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86a125b33b0a4c109a27d7626ec4568e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.036767 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 86a125b33b0a4c109a27d7626ec4568e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1407a4f230fa45469e0f6168d7549f59]), partition=
I20240501 19:56:34.037000 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86a125b33b0a4c109a27d7626ec4568e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.039076 14272 tablet_bootstrap.cc:492] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:34.039966 14272 tablet_bootstrap.cc:654] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.040776 14267 tablet_bootstrap.cc:492] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:34.041172 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 86a125b33b0a4c109a27d7626ec4568e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1407a4f230fa45469e0f6168d7549f59]), partition=
I20240501 19:56:34.041399 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86a125b33b0a4c109a27d7626ec4568e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.041620 14267 tablet_bootstrap.cc:654] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.043313 14265 tablet_bootstrap.cc:492] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:34.044106 14265 tablet_bootstrap.cc:654] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.049746 14267 tablet_bootstrap.cc:492] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:34.049844 14267 ts_tablet_manager.cc:1367] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240501 19:56:34.049901 14272 tablet_bootstrap.cc:492] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:34.049983 14272 ts_tablet_manager.cc:1367] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20240501 19:56:34.050282 14267 raft_consensus.cc:348] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.050396 14267 raft_consensus.cc:374] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.050424 14267 raft_consensus.cc:729] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.050534 14267 consensus_queue.cc:260] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.050586 14272 raft_consensus.cc:348] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.050686 14272 raft_consensus.cc:374] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.050714 14272 raft_consensus.cc:729] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.050773 14267 ts_tablet_manager.cc:1398] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:34.050819 14272 consensus_queue.cc:260] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.051041 14272 ts_tablet_manager.cc:1398] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:34.052703 14265 tablet_bootstrap.cc:492] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:34.052804 14265 ts_tablet_manager.cc:1367] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:56:34.053246 14265 raft_consensus.cc:348] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.053349 14265 raft_consensus.cc:374] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.053377 14265 raft_consensus.cc:729] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.053496 14265 consensus_queue.cc:260] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.053789 14265 ts_tablet_manager.cc:1398] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.104339 11248 consensus_queue.cc:237] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.105851 11378 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.106166 11511 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.106346 14356 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.106571 13434 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.108650 13434 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.109169 11511 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.111276 11378 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.111904 11106 catalog_manager.cc:5484] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:34.114270 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c07d9fc05254a6399afcc54b3efb8ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.121101 11248 consensus_queue.cc:237] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.122529 11378 raft_consensus.cc:1261] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.122944 14199 consensus_queue.cc:1035] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.123076 11511 raft_consensus.cc:1261] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.123594 14199 consensus_queue.cc:1035] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.125125 14199 raft_consensus.cc:2932] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.126154 11378 raft_consensus.cc:2932] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
W20240501 19:56:34.127872 11181 consensus_peers.cc:485] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c92c6475f5ed4529b1efd12b6d495a11. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.128180 11102 catalog_manager.cc:5484] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.128271 11511 raft_consensus.cc:2932] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.137634 11644 consensus_queue.cc:237] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.139076 11378 raft_consensus.cc:1261] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.139171 11248 raft_consensus.cc:1261] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.139536 14191 consensus_queue.cc:1035] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:34.139812 11580 consensus_peers.cc:485] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90d176eeeb5c4f9399f2a608cf98e319. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.139871 14191 consensus_queue.cc:1035] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.141434 14191 raft_consensus.cc:2932] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.142033 11378 raft_consensus.cc:2932] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.142174 11249 raft_consensus.cc:2932] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.145037 11099 catalog_manager.cc:5484] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.153095 11249 consensus_queue.cc:237] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:34.154735 11644 raft_consensus.cc:1261] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.155212 14199 consensus_queue.cc:1035] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.155347 11378 raft_consensus.cc:1261] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.155814 14199 consensus_queue.cc:1035] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.157588 14356 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
W20240501 19:56:34.158002 11181 consensus_peers.cc:485] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6070b62336054ef4b0444f27844b9ee1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.158109 11378 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.160343 11644 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.161375 11106 catalog_manager.cc:5484] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.168905 11511 consensus_queue.cc:237] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.170737 11378 raft_consensus.cc:1261] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.171202 13155 consensus_queue.cc:1035] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.171455 11249 raft_consensus.cc:1261] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.171875 13437 consensus_queue.cc:1035] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.173378 13155 raft_consensus.cc:2932] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.173674 11378 raft_consensus.cc:2932] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.174525 11248 raft_consensus.cc:2932] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
W20240501 19:56:34.176093 11445 consensus_peers.cc:485] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f6a547744514cd88d167a0787d8535d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.176903 11106 catalog_manager.cc:5484] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.184455 11248 consensus_queue.cc:237] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
W20240501 19:56:34.186062 11181 consensus_peers.cc:485] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0419fba851684c9b9cf0e6096de78a03. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.186122 11378 raft_consensus.cc:1261] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.186429 11511 raft_consensus.cc:1261] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.186663 13434 consensus_queue.cc:1035] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.186856 14199 consensus_queue.cc:1035] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.189117 14356 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.189541 11378 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.190826 11511 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.192243 11099 catalog_manager.cc:5484] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.201309 11644 consensus_queue.cc:237] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
W20240501 19:56:34.202780 11580 consensus_peers.cc:485] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0fb00c1acda4d98a90b12d877412045. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.202819 11378 raft_consensus.cc:1261] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.203024 11248 raft_consensus.cc:1261] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.203343 14374 consensus_queue.cc:1035] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.203564 14191 consensus_queue.cc:1035] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.205281 14374 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.205634 11248 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.207880 11378 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.208248 11102 catalog_manager.cc:5484] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.218667 11248 consensus_queue.cc:237] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:34.220281 11644 raft_consensus.cc:1261] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.220325 11511 raft_consensus.cc:1261] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:34.220711 11181 consensus_peers.cc:485] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7542b5a2a89848e5950ca8d4befc0497. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.220783 14199 consensus_queue.cc:1035] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.220992 14356 consensus_queue.cc:1035] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.222893 14199 raft_consensus.cc:2932] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.223268 11511 raft_consensus.cc:2932] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.225589 11099 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 19:56:34.225730 11099 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 19:56:34.226286 11643 raft_consensus.cc:2932] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.229844 11099 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 4 replicas of tablet 90d176eeeb5c4f9399f2a608cf98e319
I20240501 19:56:34.230178 11099 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 7542b5a2a89848e5950ca8d4befc0497
I20240501 19:56:34.230387 11099 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet b9b18195711b48f099a29c006cd89ce5
I20240501 19:56:34.230517 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 90d176eeeb5c4f9399f2a608cf98e319 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:34.230573 13781 tablet_service.cc:1514] Processing DeleteTablet for tablet 90d176eeeb5c4f9399f2a608cf98e319 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:34.230907 14259 tablet_replica.cc:330] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:34.231004 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542b5a2a89848e5950ca8d4befc0497 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:34.231065 14259 raft_consensus.cc:2229] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.231199 14260 tablet_replica.cc:330] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:34.231233 14259 raft_consensus.cc:2258] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.231328 14260 raft_consensus.cc:2229] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.231498 14260 raft_consensus.cc:2258] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 19:56:34.231786 11092 catalog_manager.cc:4797] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): delete failed for tablet 90d176eeeb5c4f9399f2a608cf98e319 because the tablet was not found. No further retry: Not found: Tablet not found: 90d176eeeb5c4f9399f2a608cf98e319
I20240501 19:56:34.232012 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet 90d176eeeb5c4f9399f2a608cf98e319 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:34.232085 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b18195711b48f099a29c006cd89ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:34.232193 14261 tablet_replica.cc:330] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:34.232197 14260 ts_tablet_manager.cc:1875] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.232316 14261 raft_consensus.cc:2229] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.232479 14261 raft_consensus.cc:2258] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.232836 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542b5a2a89848e5950ca8d4befc0497 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:34.232968 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b18195711b48f099a29c006cd89ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:34.233072 11622 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542b5a2a89848e5950ca8d4befc0497 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:34.233144 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b18195711b48f099a29c006cd89ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:34.233510 14261 ts_tablet_manager.cc:1875] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.233956 14259 ts_tablet_manager.cc:1875] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.234356 14260 ts_tablet_manager.cc:1888] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:34.234414 14260 log.cc:1196] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/7542b5a2a89848e5950ca8d4befc0497
I20240501 19:56:34.234635 14260 ts_tablet_manager.cc:1909] T 7542b5a2a89848e5950ca8d4befc0497 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:34.234813 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 90d176eeeb5c4f9399f2a608cf98e319 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:34.234966 14258 tablet_replica.cc:330] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:34.235091 14258 raft_consensus.cc:2229] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.235242 14258 raft_consensus.cc:2258] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.235412 14261 ts_tablet_manager.cc:1888] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:34.235504 14261 log.cc:1196] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/90d176eeeb5c4f9399f2a608cf98e319
I20240501 19:56:34.235684 14261 ts_tablet_manager.cc:1909] T 90d176eeeb5c4f9399f2a608cf98e319 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:34.235958 14258 ts_tablet_manager.cc:1875] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.236129 14259 ts_tablet_manager.cc:1888] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:34.236186 14259 log.cc:1196] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/90d176eeeb5c4f9399f2a608cf98e319
I20240501 19:56:34.236399 14259 ts_tablet_manager.cc:1909] T 90d176eeeb5c4f9399f2a608cf98e319 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:34.236400 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 90d176eeeb5c4f9399f2a608cf98e319 (table rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]) successfully deleted
I20240501 19:56:34.236311 14261 tablet_replica.cc:330] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:34.236619 14261 raft_consensus.cc:2229] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.236709 14261 raft_consensus.cc:2258] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.237051 14261 ts_tablet_manager.cc:1875] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.237637 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 7542b5a2a89848e5950ca8d4befc0497 (table rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]) successfully deleted
I20240501 19:56:34.237782 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 90d176eeeb5c4f9399f2a608cf98e319 (table rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]) successfully deleted
I20240501 19:56:34.237987 14258 ts_tablet_manager.cc:1888] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:34.238050 14258 log.cc:1196] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/7542b5a2a89848e5950ca8d4befc0497
I20240501 19:56:34.238270 14258 ts_tablet_manager.cc:1909] T 7542b5a2a89848e5950ca8d4befc0497 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:34.238574 14261 ts_tablet_manager.cc:1888] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.238636 14261 log.cc:1196] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/b9b18195711b48f099a29c006cd89ce5
I20240501 19:56:34.238862 14261 ts_tablet_manager.cc:1909] T b9b18195711b48f099a29c006cd89ce5 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:34.239688 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet b9b18195711b48f099a29c006cd89ce5 (table rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]) successfully deleted
I20240501 19:56:34.240479 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 7542b5a2a89848e5950ca8d4befc0497 (table rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]) successfully deleted
I20240501 19:56:34.240600 14258 tablet_replica.cc:330] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:34.240733 14258 raft_consensus.cc:2229] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:34.240907 14258 raft_consensus.cc:2258] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.241291 14259 tablet_replica.cc:330] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:34.241966 14258 ts_tablet_manager.cc:1875] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.242707 14259 raft_consensus.cc:2229] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:34.243054 14259 raft_consensus.cc:2258] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.243726 14259 ts_tablet_manager.cc:1875] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.244935 14374 raft_consensus.cc:482] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:34.245033 14374 raft_consensus.cc:504] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.245417 14374 leader_election.cc:288] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:34.245776 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86a125b33b0a4c109a27d7626ec4568e" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:34.245811 13670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86a125b33b0a4c109a27d7626ec4568e" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:34.245904 11248 raft_consensus.cc:2454] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:34.245929 13670 raft_consensus.cc:2454] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:34.246213 11577 leader_election.cc:302] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:34.246495 11511 consensus_queue.cc:237] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.247848 14258 ts_tablet_manager.cc:1888] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.247931 14258 log.cc:1196] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/b9b18195711b48f099a29c006cd89ce5
I20240501 19:56:34.248095 11378 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:34.248250 11445 consensus_peers.cc:485] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e63472cfff449089d88d18b9e495912. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.248299 14258 ts_tablet_manager.cc:1909] T b9b18195711b48f099a29c006cd89ce5 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:34.248476 13437 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.248843 14258 tablet_replica.cc:330] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:34.249101 14259 ts_tablet_manager.cc:1888] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:34.249179 14259 log.cc:1196] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/7542b5a2a89848e5950ca8d4befc0497
I20240501 19:56:34.249428 14259 ts_tablet_manager.cc:1909] T 7542b5a2a89848e5950ca8d4befc0497 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:34.249840 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet b9b18195711b48f099a29c006cd89ce5 (table rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]) successfully deleted
I20240501 19:56:34.250005 14259 tablet_replica.cc:330] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:34.250051 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 7542b5a2a89848e5950ca8d4befc0497 (table rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]) successfully deleted
I20240501 19:56:34.250152 14259 raft_consensus.cc:2229] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.248943 14374 raft_consensus.cc:2781] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:34.250293 14374 raft_consensus.cc:482] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:34.250355 14374 raft_consensus.cc:3037] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.250664 13155 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.251060 14374 raft_consensus.cc:504] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.251092 11378 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.251416 14374 leader_election.cc:288] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:34.249137 11643 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.252246 14258 raft_consensus.cc:2229] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:34.252477 14258 raft_consensus.cc:2258] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.251897 11248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86a125b33b0a4c109a27d7626ec4568e" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:34.252691 11248 raft_consensus.cc:3037] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.252754 13670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86a125b33b0a4c109a27d7626ec4568e" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:34.252864 13670 raft_consensus.cc:3037] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.253171 13437 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.253592 11248 raft_consensus.cc:2454] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:34.253743 13670 raft_consensus.cc:2454] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:34.254752 11102 catalog_manager.cc:5484] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.255576 11577 leader_election.cc:302] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:34.255844 14191 raft_consensus.cc:2781] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:34.255925 14191 raft_consensus.cc:686] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:34.256057 14191 consensus_queue.cc:237] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.256419 11643 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.258124 13150 consensus_queue.cc:1035] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.258275 14258 ts_tablet_manager.cc:1875] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.258738 11101 catalog_manager.cc:5484] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.260277 14258 ts_tablet_manager.cc:1888] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:34.260351 14258 log.cc:1196] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/90d176eeeb5c4f9399f2a608cf98e319
I20240501 19:56:34.260573 14258 ts_tablet_manager.cc:1909] T 90d176eeeb5c4f9399f2a608cf98e319 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:34.260794 14259 raft_consensus.cc:2258] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.261368 14259 ts_tablet_manager.cc:1875] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.264222 14259 ts_tablet_manager.cc:1888] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.263695 14191 consensus_queue.cc:1035] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.264670 14259 log.cc:1196] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/b9b18195711b48f099a29c006cd89ce5
I20240501 19:56:34.265700 14259 ts_tablet_manager.cc:1909] T b9b18195711b48f099a29c006cd89ce5 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:34.267040 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet b9b18195711b48f099a29c006cd89ce5 (table rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]) successfully deleted
I20240501 19:56:34.269588 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 90d176eeeb5c4f9399f2a608cf98e319 (table rebalance_test_table_2 [id=95a9b7c163014da6ae649d8848c34817]) successfully deleted
I20240501 19:56:34.271996 11643 consensus_queue.cc:237] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.273707 11511 raft_consensus.cc:1261] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.274154 11248 raft_consensus.cc:1261] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.274202 14191 consensus_queue.cc:1035] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.274503 14374 consensus_queue.cc:1035] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.276540 14374 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.278882 11511 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.279469 11101 catalog_manager.cc:5484] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:34.280807 11577 consensus_peers.cc:485] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2ecce0ca7a941dbb98b89b797beb8a1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.281107 11248 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.287847 11643 consensus_queue.cc:237] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.289299 11378 raft_consensus.cc:1261] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.289489 11248 raft_consensus.cc:1261] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.289707 14374 consensus_queue.cc:1035] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.289901 13150 consensus_queue.cc:1035] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:34.290103 11580 consensus_peers.cc:485] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 582d72e630c84088a6e4a7309ae7e952. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.291527 14374 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.292040 11248 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.292119 11378 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.295084 11102 catalog_manager.cc:5484] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.302718 11511 consensus_queue.cc:237] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:34.304152 11643 raft_consensus.cc:1261] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.304481 11248 raft_consensus.cc:1261] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.304700 13437 consensus_queue.cc:1035] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.304898 14391 consensus_queue.cc:1035] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.306746 13437 raft_consensus.cc:2932] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.307090 11643 raft_consensus.cc:2932] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
W20240501 19:56:34.309012 11445 consensus_peers.cc:485] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea873ec8ced340b4ac082417c9b09aae. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.309446 11248 raft_consensus.cc:2932] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.309971 11101 catalog_manager.cc:5484] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.318192 11643 consensus_queue.cc:237] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.319514 11378 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.319650 11511 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:34.319733 11577 consensus_peers.cc:485] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e307d45fb57f444ca09d94d59b9bcd64. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.319983 14191 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.320168 13150 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.321903 14374 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.322335 11378 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.322451 11511 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.325029 11102 catalog_manager.cc:5484] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.333801 11378 consensus_queue.cc:237] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.335317 11643 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.335609 11511 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.335763 14151 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.335988 14473 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:34.339102 11313 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cebd15701ee44b11a2f8804740fb4299. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.339049 13209 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.339615 11511 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.341245 11643 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.342096 11099 catalog_manager.cc:5484] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.351424 11511 consensus_queue.cc:237] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.352934 11643 raft_consensus.cc:1261] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.353408 13437 consensus_queue.cc:1035] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.356092 13155 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.356541 11643 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.357131 11248 raft_consensus.cc:1261] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.357734 13155 consensus_queue.cc:1035] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:34.358062 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04932055883c412f944e839982815ab6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.358989 11248 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.359560 11101 catalog_manager.cc:5484] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.369990 11378 consensus_queue.cc:237] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.371846 11643 raft_consensus.cc:1261] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.372051 11248 raft_consensus.cc:1261] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.372292 14473 consensus_queue.cc:1035] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.373178 14473 consensus_queue.cc:1035] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.373461 14332 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.376376 14473 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.377373 11248 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.379623 14129 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:34.380107 11313 consensus_peers.cc:485] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f1f6382bcc84fcda444dcfae87b00c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.380110 11099 catalog_manager.cc:5484] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.389065 11099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:34.391606 11643 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:34.394032 13670 raft_consensus.cc:1261] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.394064 11248 raft_consensus.cc:1261] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:34.394071 11643 consensus_queue.cc:237] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:34.394483 13150 consensus_queue.cc:1035] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.394637 14374 consensus_queue.cc:1035] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.395493 11378 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.395890 11511 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.396061 14374 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.396833 13150 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.398689 14505 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.399492 11101 catalog_manager.cc:5576] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501) does not have the latest schema for tablet 86a125b33b0a4c109a27d7626ec4568e (table rebalancer_dynamic_table [id=1407a4f230fa45469e0f6168d7549f59]). Expected version 1 got 0
I20240501 19:56:34.400138 11099 catalog_manager.cc:5576] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965) does not have the latest schema for tablet 86a125b33b0a4c109a27d7626ec4568e (table rebalancer_dynamic_table [id=1407a4f230fa45469e0f6168d7549f59]). Expected version 1 got 0
I20240501 19:56:34.401014 11378 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
W20240501 19:56:34.402721 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69facf86bba24f209c06afafc1ba8f7b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.404194 11099 catalog_manager.cc:5484] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.404650 14435 mvcc.cc:205] Tried to move back new op lower bound from 7022974543432626176 to 7022974542874202112. Current Snapshot: MvccSnapshot[applied={T|T < 7022974543432626176}]
I20240501 19:56:34.404774 14515 tablet.cc:1722] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:34.405550 14515 tablet.cc:1620] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: MemRowSet was empty: no flush needed.
I20240501 19:56:34.406481 11106 catalog_manager.cc:5576] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501) does not have the latest schema for tablet 86a125b33b0a4c109a27d7626ec4568e (table rebalancer_dynamic_table [id=1407a4f230fa45469e0f6168d7549f59]). Expected version 1 got 0
I20240501 19:56:34.409561 14517 tablet.cc:1722] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:34.410770 14517 tablet.cc:1620] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: MemRowSet was empty: no flush needed.
I20240501 19:56:34.411054 11099 catalog_manager.cc:5576] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965) does not have the latest schema for tablet 86a125b33b0a4c109a27d7626ec4568e (table rebalancer_dynamic_table [id=1407a4f230fa45469e0f6168d7549f59]). Expected version 1 got 0
I20240501 19:56:34.412147 14516 tablet.cc:1722] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:34.412895 11101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=1407a4f230fa45469e0f6168d7549f59] alter complete (version 1)
I20240501 19:56:34.413491 14516 tablet.cc:1620] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:34.415966 11511 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:34.421826 11378 consensus_queue.cc:237] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.423352 11511 raft_consensus.cc:1261] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:34.423492 11313 consensus_peers.cc:485] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70fcac1c11404f51838dcb000f76d737. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.423698 11248 raft_consensus.cc:1261] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.423895 14151 consensus_queue.cc:1035] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.424058 14477 consensus_queue.cc:1035] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.426175 13209 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.426673 11511 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.426671 11248 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.429296 11101 catalog_manager.cc:5484] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:34.437762 11643 consensus_queue.cc:237] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.439083 11248 raft_consensus.cc:1261] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:34.439291 11511 raft_consensus.cc:1261] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:34.439344 11580 consensus_peers.cc:485] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d99f4d3127348dba949ab5fd4666956. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:34.439509 14505 consensus_queue.cc:1035] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.439713 14374 consensus_queue.cc:1035] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.441591 14191 raft_consensus.cc:2932] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.442122 11511 raft_consensus.cc:2932] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.442130 11249 raft_consensus.cc:2932] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:34.445150 11106 catalog_manager.cc:5484] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.494277 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:34.500576 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 96511455168148b7b619670f635d3a3e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.500871 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96511455168148b7b619670f635d3a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.500839 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 96511455168148b7b619670f635d3a3e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.501109 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96511455168148b7b619670f635d3a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.501838 13912 tablet_service.cc:1467] Processing CreateTablet for tablet dcb07fd6b982437195ff42e5bf0c48be (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.501935 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 96511455168148b7b619670f635d3a3e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.502094 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb07fd6b982437195ff42e5bf0c48be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.502208 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96511455168148b7b619670f635d3a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.503141 14272 tablet_bootstrap.cc:492] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:34.503640 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 22e166cb5c864fd993c67c5c08cab1d0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.503916 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e166cb5c864fd993c67c5c08cab1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.504217 14267 tablet_bootstrap.cc:492] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:34.504309 14272 tablet_bootstrap.cc:654] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.504642 13911 tablet_service.cc:1467] Processing CreateTablet for tablet 22e166cb5c864fd993c67c5c08cab1d0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.504860 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e166cb5c864fd993c67c5c08cab1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.505180 14267 tablet_bootstrap.cc:654] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.506209 14584 tablet_bootstrap.cc:492] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:34.506335 13781 tablet_service.cc:1467] Processing CreateTablet for tablet dcb07fd6b982437195ff42e5bf0c48be (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.506628 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb07fd6b982437195ff42e5bf0c48be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.507228 14584 tablet_bootstrap.cc:654] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.508230 14580 tablet_bootstrap.cc:492] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:34.508384 14584 tablet_bootstrap.cc:492] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:34.508383 14272 tablet_bootstrap.cc:492] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:34.508466 14272 ts_tablet_manager.cc:1367] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:34.508466 14584 ts_tablet_manager.cc:1367] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:34.508688 13650 tablet_service.cc:1467] Processing CreateTablet for tablet dcb07fd6b982437195ff42e5bf0c48be (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.508915 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb07fd6b982437195ff42e5bf0c48be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.508919 14584 raft_consensus.cc:348] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.509024 14584 raft_consensus.cc:374] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.509055 14584 raft_consensus.cc:729] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.509169 14584 consensus_queue.cc:260] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.509258 14580 tablet_bootstrap.cc:654] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.509454 14584 ts_tablet_manager.cc:1398] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.511080 14267 tablet_bootstrap.cc:492] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:34.511168 14267 ts_tablet_manager.cc:1367] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 19:56:34.511117 13649 tablet_service.cc:1467] Processing CreateTablet for tablet 22e166cb5c864fd993c67c5c08cab1d0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:34.511363 13649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e166cb5c864fd993c67c5c08cab1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.511718 14267 raft_consensus.cc:348] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.511826 14267 raft_consensus.cc:374] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.511855 14267 raft_consensus.cc:729] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.511970 14267 consensus_queue.cc:260] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.512216 14267 ts_tablet_manager.cc:1398] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.514014 14580 tablet_bootstrap.cc:492] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:34.514120 14580 ts_tablet_manager.cc:1367] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:34.508919 14272 raft_consensus.cc:348] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.514489 14272 raft_consensus.cc:374] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.514533 14272 raft_consensus.cc:729] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.514660 14272 consensus_queue.cc:260] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.514712 14580 raft_consensus.cc:348] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.514797 14580 raft_consensus.cc:374] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.514824 14580 raft_consensus.cc:729] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.514942 14272 ts_tablet_manager.cc:1398] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:34.514967 14580 consensus_queue.cc:260] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.515223 14272 tablet_bootstrap.cc:492] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:34.515758 14580 ts_tablet_manager.cc:1398] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:34.515897 14580 tablet_bootstrap.cc:492] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:34.516139 14272 tablet_bootstrap.cc:654] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.516790 14580 tablet_bootstrap.cc:654] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.517882 14272 tablet_bootstrap.cc:492] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:34.517963 14272 ts_tablet_manager.cc:1367] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:34.518357 14272 raft_consensus.cc:348] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.518446 14272 raft_consensus.cc:374] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.518476 14272 raft_consensus.cc:729] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.518580 14272 consensus_queue.cc:260] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.518824 14272 ts_tablet_manager.cc:1398] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.518993 14272 tablet_bootstrap.cc:492] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:34.519026 11106 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:34.519887 14272 tablet_bootstrap.cc:654] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.519891 14581 tablet_bootstrap.cc:492] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:34.520814 14581 tablet_bootstrap.cc:654] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.521570 14587 tablet_bootstrap.cc:492] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:34.522459 14587 tablet_bootstrap.cc:654] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.523528 14581 tablet_bootstrap.cc:492] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:34.523677 14581 ts_tablet_manager.cc:1367] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:34.523730 14272 tablet_bootstrap.cc:492] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:34.523818 14272 ts_tablet_manager.cc:1367] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:34.524222 14272 raft_consensus.cc:348] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.524330 14272 raft_consensus.cc:374] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.524367 14272 raft_consensus.cc:729] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.524475 14272 consensus_queue.cc:260] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.524525 14581 raft_consensus.cc:348] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.524725 14581 raft_consensus.cc:374] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.524745 14272 ts_tablet_manager.cc:1398] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:34.524879 14581 raft_consensus.cc:729] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.525079 14581 consensus_queue.cc:260] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.525383 14581 ts_tablet_manager.cc:1398] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:34.525854 14580 tablet_bootstrap.cc:492] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:34.526576 14587 tablet_bootstrap.cc:492] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:34.526796 14580 ts_tablet_manager.cc:1367] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:34.526863 14587 ts_tablet_manager.cc:1367] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:34.527230 14580 raft_consensus.cc:348] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.527318 14580 raft_consensus.cc:374] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.527271 14587 raft_consensus.cc:348] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.527345 14580 raft_consensus.cc:729] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.527371 14587 raft_consensus.cc:374] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.527401 14587 raft_consensus.cc:729] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.527526 14580 consensus_queue.cc:260] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.527782 14587 consensus_queue.cc:260] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.527904 14580 ts_tablet_manager.cc:1398] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.528607 14587 ts_tablet_manager.cc:1398] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:34.529630 14515 tablet.cc:1722] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:34.529947 14515 tablet.cc:1620] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: MemRowSet was empty: no flush needed.
I20240501 19:56:34.530588 14517 tablet.cc:1722] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:34.530866 14517 tablet.cc:1620] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: MemRowSet was empty: no flush needed.
I20240501 19:56:34.531617 14516 tablet.cc:1722] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:34.531898 14516 tablet.cc:1620] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:34.536763 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=1407a4f230fa45469e0f6168d7549f59] alter complete (version 2)
I20240501 19:56:34.538966 14598 ts_tablet_manager.cc:902] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:34.539973 14598 tablet_copy_client.cc:308] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:42965
I20240501 19:56:34.541062 11269 tablet_copy_service.cc:140] P 10ee60ead7b94c93ab749ec16a04b0dc: Received BeginTabletCopySession request for tablet 6070b62336054ef4b0444f27844b9ee1 from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:42125)
I20240501 19:56:34.541201 11269 tablet_copy_service.cc:161] P 10ee60ead7b94c93ab749ec16a04b0dc: Beginning new tablet copy session on tablet 6070b62336054ef4b0444f27844b9ee1 from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:42125: session id = 6921cf2e41f64d66842885512da8de6c-6070b62336054ef4b0444f27844b9ee1
I20240501 19:56:34.542513 11269 tablet_copy_source_session.cc:205] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.542912 11378 raft_consensus.cc:3037] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.545604 14129 consensus_queue.cc:1035] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.546543 14598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6070b62336054ef4b0444f27844b9ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.549978 14280 raft_consensus.cc:482] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:34.550076 14280 raft_consensus.cc:504] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.548928 14598 tablet_copy_client.cc:783] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.550896 14598 tablet_copy_client.cc:647] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.551108 14280 leader_election.cc:288] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:34.551852 14129 consensus_queue.cc:1035] T d7bb9ef27c6e4f248a0e5a4812986f38 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:34.554301 14603 ts_tablet_manager.cc:902] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:34.555495 14603 tablet_copy_client.cc:308] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:42965
I20240501 19:56:34.558375 14598 tablet_copy_client.cc:523] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.558669 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e166cb5c864fd993c67c5c08cab1d0" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:34.558804 13932 raft_consensus.cc:2454] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:34.559196 13603 leader_election.cc:302] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:34.559497 14280 raft_consensus.cc:2781] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:34.559549 14280 raft_consensus.cc:482] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:34.559581 14280 raft_consensus.cc:3037] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.560159 14598 tablet_bootstrap.cc:492] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:34.560544 14280 raft_consensus.cc:504] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.560926 14280 leader_election.cc:288] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:34.561645 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e166cb5c864fd993c67c5c08cab1d0" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:34.561748 13932 raft_consensus.cc:3037] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.562630 13932 raft_consensus.cc:2454] T 22e166cb5c864fd993c67c5c08cab1d0 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:34.563356 13603 leader_election.cc:302] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:34.563598 14280 raft_consensus.cc:2781] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:34.563761 14280 raft_consensus.cc:686] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:34.564067 14280 consensus_queue.cc:237] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.566552 11099 catalog_manager.cc:5484] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.566905 11269 tablet_copy_service.cc:140] P 10ee60ead7b94c93ab749ec16a04b0dc: Received BeginTabletCopySession request for tablet c92c6475f5ed4529b1efd12b6d495a11 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:50213)
I20240501 19:56:34.567034 11269 tablet_copy_service.cc:161] P 10ee60ead7b94c93ab749ec16a04b0dc: Beginning new tablet copy session on tablet c92c6475f5ed4529b1efd12b6d495a11 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:50213: session id = 533908e517b24f77b2af8b5863e4ae5a-c92c6475f5ed4529b1efd12b6d495a11
I20240501 19:56:34.568012 11269 tablet_copy_source_session.cc:205] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.569891 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92c6475f5ed4529b1efd12b6d495a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.572098 14603 tablet_copy_client.cc:783] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.572317 14603 tablet_copy_client.cc:647] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.573107 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e166cb5c864fd993c67c5c08cab1d0" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:34.573256 11378 raft_consensus.cc:2454] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:34.573472 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e166cb5c864fd993c67c5c08cab1d0" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:34.573578 11377 raft_consensus.cc:3037] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.574584 11377 raft_consensus.cc:2454] T 22e166cb5c864fd993c67c5c08cab1d0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:34.580710 14603 tablet_copy_client.cc:523] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.581262 14622 ts_tablet_manager.cc:902] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:34.582106 14622 tablet_copy_client.cc:308] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:34.582589 14603 tablet_bootstrap.cc:492] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:34.591895 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 1f6a547744514cd88d167a0787d8535d from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:53821)
I20240501 19:56:34.592047 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 1f6a547744514cd88d167a0787d8535d from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:53821: session id = 8125d8ad6d0d44caae9c5c90c90df96b-1f6a547744514cd88d167a0787d8535d
I20240501 19:56:34.593438 11533 tablet_copy_source_session.cc:205] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.595487 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6a547744514cd88d167a0787d8535d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.597791 14622 tablet_copy_client.cc:783] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.597959 14622 tablet_copy_client.cc:647] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.599808 14622 tablet_copy_client.cc:523] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.601330 14622 tablet_bootstrap.cc:492] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:34.605603 14598 tablet_bootstrap.cc:492] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.606003 14598 tablet_bootstrap.cc:492] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:34.606238 14598 ts_tablet_manager.cc:1367] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.046s user 0.028s sys 0.002s
I20240501 19:56:34.607206 14598 raft_consensus.cc:348] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:34.607481 14598 raft_consensus.cc:729] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:34.607642 14598 consensus_queue.cc:260] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:34.608561 14598 ts_tablet_manager.cc:1398] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:34.609407 11269 tablet_copy_service.cc:342] P 10ee60ead7b94c93ab749ec16a04b0dc: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-6070b62336054ef4b0444f27844b9ee1 received from {username='slave'} at 127.5.40.7:42125
I20240501 19:56:34.609504 11269 tablet_copy_service.cc:433] P 10ee60ead7b94c93ab749ec16a04b0dc: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-6070b62336054ef4b0444f27844b9ee1 on tablet 6070b62336054ef4b0444f27844b9ee1 with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:34.613756 14603 tablet_bootstrap.cc:492] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.614140 14639 ts_tablet_manager.cc:902] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:34.614161 14603 tablet_bootstrap.cc:492] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:34.614414 14603 ts_tablet_manager.cc:1367] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.004s
I20240501 19:56:34.614711 14639 tablet_copy_client.cc:308] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:42965
I20240501 19:56:34.614902 14603 raft_consensus.cc:348] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.615041 14603 raft_consensus.cc:729] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:34.615175 14603 consensus_queue.cc:260] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.615203 11269 tablet_copy_service.cc:140] P 10ee60ead7b94c93ab749ec16a04b0dc: Received BeginTabletCopySession request for tablet 0419fba851684c9b9cf0e6096de78a03 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:50213)
I20240501 19:56:34.615300 11269 tablet_copy_service.cc:161] P 10ee60ead7b94c93ab749ec16a04b0dc: Beginning new tablet copy session on tablet 0419fba851684c9b9cf0e6096de78a03 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:50213: session id = 533908e517b24f77b2af8b5863e4ae5a-0419fba851684c9b9cf0e6096de78a03
I20240501 19:56:34.616118 14603 ts_tablet_manager.cc:1398] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:34.616299 11269 tablet_copy_source_session.cc:205] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.617308 14639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0419fba851684c9b9cf0e6096de78a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.617373 11269 tablet_copy_service.cc:342] P 10ee60ead7b94c93ab749ec16a04b0dc: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-c92c6475f5ed4529b1efd12b6d495a11 received from {username='slave'} at 127.5.40.8:50213
I20240501 19:56:34.617427 11269 tablet_copy_service.cc:433] P 10ee60ead7b94c93ab749ec16a04b0dc: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-c92c6475f5ed4529b1efd12b6d495a11 on tablet c92c6475f5ed4529b1efd12b6d495a11 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:34.619825 14639 tablet_copy_client.cc:783] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.619964 14639 tablet_copy_client.cc:647] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.621183 14639 tablet_copy_client.cc:523] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.622817 14639 tablet_bootstrap.cc:492] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:34.638516 11106 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:34.641511 14622 tablet_bootstrap.cc:492] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.641906 14622 tablet_bootstrap.cc:492] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:34.642263 14622 ts_tablet_manager.cc:1367] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.041s user 0.031s sys 0.000s
I20240501 19:56:34.642804 14622 raft_consensus.cc:348] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.643034 14622 raft_consensus.cc:729] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:34.643234 14622 consensus_queue.cc:260] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.643700 14622 ts_tablet_manager.cc:1398] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.644546 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-1f6a547744514cd88d167a0787d8535d received from {username='slave'} at 127.5.40.5:53821
I20240501 19:56:34.644647 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-1f6a547744514cd88d167a0787d8535d on tablet 1f6a547744514cd88d167a0787d8535d with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:34.644933 14515 tablet.cc:1722] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:34.645735 14516 tablet.cc:1722] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:34.647753 14517 tablet.cc:1722] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:34.655803 11099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=1407a4f230fa45469e0f6168d7549f59] alter complete (version 3)
I20240501 19:56:34.659751 14622 ts_tablet_manager.cc:902] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:34.660202 14622 tablet_copy_client.cc:308] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:42965
I20240501 19:56:34.664064 14639 tablet_bootstrap.cc:492] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.664561 14639 tablet_bootstrap.cc:492] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:34.664685 11269 tablet_copy_service.cc:140] P 10ee60ead7b94c93ab749ec16a04b0dc: Received BeginTabletCopySession request for tablet 1c07d9fc05254a6399afcc54b3efb8ac from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:37827)
I20240501 19:56:34.664784 11269 tablet_copy_service.cc:161] P 10ee60ead7b94c93ab749ec16a04b0dc: Beginning new tablet copy session on tablet 1c07d9fc05254a6399afcc54b3efb8ac from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:37827: session id = 8125d8ad6d0d44caae9c5c90c90df96b-1c07d9fc05254a6399afcc54b3efb8ac
I20240501 19:56:34.665658 11269 tablet_copy_source_session.cc:205] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.666172 14639 ts_tablet_manager.cc:1367] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.044s user 0.025s sys 0.002s
I20240501 19:56:34.666652 14639 raft_consensus.cc:348] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.666779 14639 raft_consensus.cc:729] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:34.666906 14639 consensus_queue.cc:260] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.667260 14639 ts_tablet_manager.cc:1398] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.667789 11269 tablet_copy_service.cc:342] P 10ee60ead7b94c93ab749ec16a04b0dc: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-0419fba851684c9b9cf0e6096de78a03 received from {username='slave'} at 127.5.40.8:50213
I20240501 19:56:34.667858 11269 tablet_copy_service.cc:433] P 10ee60ead7b94c93ab749ec16a04b0dc: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-0419fba851684c9b9cf0e6096de78a03 on tablet 0419fba851684c9b9cf0e6096de78a03 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:34.669294 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c07d9fc05254a6399afcc54b3efb8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.671383 14622 tablet_copy_client.cc:783] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.672434 14622 tablet_copy_client.cc:647] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.674364 14622 tablet_copy_client.cc:523] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.675668 14622 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:34.682067 14672 ts_tablet_manager.cc:902] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:34.683107 14672 tablet_copy_client.cc:308] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:34.687690 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet f0fb00c1acda4d98a90b12d877412045 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:37783)
I20240501 19:56:34.687832 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet f0fb00c1acda4d98a90b12d877412045 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:37783: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-f0fb00c1acda4d98a90b12d877412045
I20240501 19:56:34.689230 11664 tablet_copy_source_session.cc:205] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.691176 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fb00c1acda4d98a90b12d877412045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.693378 14672 tablet_copy_client.cc:783] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.693539 14672 tablet_copy_client.cc:647] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.695242 14672 tablet_copy_client.cc:523] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.696712 14672 tablet_bootstrap.cc:492] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:34.718060 14618 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:34.718178 14618 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.718647 14618 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:34.720966 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:34.721105 13932 raft_consensus.cc:2454] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:34.721858 13603 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:34.724884 14618 raft_consensus.cc:2781] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:34.724962 14618 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:34.725000 14618 raft_consensus.cc:3037] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.726037 14618 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.726405 14618 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:34.727362 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:34.727516 13932 raft_consensus.cc:3037] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.728536 13932 raft_consensus.cc:2454] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:34.729573 13603 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:34.731194 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:34.731237 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:34.731324 13801 raft_consensus.cc:2454] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:34.732528 14672 tablet_bootstrap.cc:492] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.732836 14618 raft_consensus.cc:2781] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:34.732942 14618 raft_consensus.cc:686] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:34.733081 14618 consensus_queue.cc:237] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:34.733220 14672 tablet_bootstrap.cc:492] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:34.733467 14672 ts_tablet_manager.cc:1367] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.000s
I20240501 19:56:34.733938 14672 raft_consensus.cc:348] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.734093 14672 raft_consensus.cc:729] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:34.734231 14672 consensus_queue.cc:260] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.734544 11103 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 19:56:34.734664 11103 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 19:56:34.734915 11106 catalog_manager.cc:5484] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.738121 11103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 9e327331ed1a4790af392bf83e6a7cd2
I20240501 19:56:34.738420 11103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet fc32d7a811f741f4927b60b0bb9b6046
I20240501 19:56:34.738677 11103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 8fc636fe78c14563aac456322e1eda5c
I20240501 19:56:34.738709 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e327331ed1a4790af392bf83e6a7cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:34.738829 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet fc32d7a811f741f4927b60b0bb9b6046 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:34.738962 14261 tablet_replica.cc:330] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:34.739115 14261 raft_consensus.cc:2229] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.739156 14258 tablet_replica.cc:330] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:34.739208 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc636fe78c14563aac456322e1eda5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:34.739257 14261 raft_consensus.cc:2258] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.739298 14258 raft_consensus.cc:2229] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.739430 14258 raft_consensus.cc:2258] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.739846 14261 ts_tablet_manager.cc:1875] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.739907 14672 ts_tablet_manager.cc:1398] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:34.739962 14258 ts_tablet_manager.cc:1875] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.740139 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e327331ed1a4790af392bf83e6a7cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:34.740490 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-f0fb00c1acda4d98a90b12d877412045 received from {username='slave'} at 127.5.40.6:37783
I20240501 19:56:34.740557 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-f0fb00c1acda4d98a90b12d877412045 on tablet f0fb00c1acda4d98a90b12d877412045 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:34.741575 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc636fe78c14563aac456322e1eda5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:34.741786 14261 ts_tablet_manager.cc:1888] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.741863 14261 log.cc:1196] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/9e327331ed1a4790af392bf83e6a7cd2
I20240501 19:56:34.742062 14261 ts_tablet_manager.cc:1909] T 9e327331ed1a4790af392bf83e6a7cd2 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:34.742065 14258 ts_tablet_manager.cc:1888] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.742125 14258 log.cc:1196] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/fc32d7a811f741f4927b60b0bb9b6046
I20240501 19:56:34.742293 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e327331ed1a4790af392bf83e6a7cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:34.742508 14259 tablet_replica.cc:330] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:34.742653 14259 raft_consensus.cc:2229] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.742792 14259 raft_consensus.cc:2258] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.742862 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet fc32d7a811f741f4927b60b0bb9b6046 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:34.743117 11228 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc636fe78c14563aac456322e1eda5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:34.743270 14259 ts_tablet_manager.cc:1875] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.745517 14259 ts_tablet_manager.cc:1888] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.745582 14259 log.cc:1196] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/9e327331ed1a4790af392bf83e6a7cd2
I20240501 19:56:34.745818 14259 ts_tablet_manager.cc:1909] T 9e327331ed1a4790af392bf83e6a7cd2 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:34.746397 14259 tablet_replica.cc:330] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:34.747133 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 9e327331ed1a4790af392bf83e6a7cd2 (table rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]) successfully deleted
I20240501 19:56:34.747272 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet fc32d7a811f741f4927b60b0bb9b6046 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:34.746518 14259 raft_consensus.cc:2229] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:34.747705 14259 raft_consensus.cc:2258] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.747160 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 9e327331ed1a4790af392bf83e6a7cd2 (table rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]) successfully deleted
I20240501 19:56:34.748175 14598 ts_tablet_manager.cc:902] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:34.748229 14259 ts_tablet_manager.cc:1875] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.748637 14598 tablet_copy_client.cc:308] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:34.748075 14261 tablet_replica.cc:330] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:34.750720 14261 raft_consensus.cc:2229] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.751031 14261 raft_consensus.cc:2258] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.751996 14261 ts_tablet_manager.cc:1875] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.752017 14258 ts_tablet_manager.cc:1909] T fc32d7a811f741f4927b60b0bb9b6046 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:34.752827 14258 tablet_replica.cc:330] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:34.752887 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet fc32d7a811f741f4927b60b0bb9b6046 (table rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]) successfully deleted
I20240501 19:56:34.752975 14258 raft_consensus.cc:2229] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:34.753211 14258 raft_consensus.cc:2258] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.753756 14258 ts_tablet_manager.cc:1875] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.755838 14258 ts_tablet_manager.cc:1888] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.755980 14258 log.cc:1196] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/8fc636fe78c14563aac456322e1eda5c
I20240501 19:56:34.756325 14258 ts_tablet_manager.cc:1909] T 8fc636fe78c14563aac456322e1eda5c P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:34.757999 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 8fc636fe78c14563aac456322e1eda5c (table rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]) successfully deleted
I20240501 19:56:34.758662 14259 ts_tablet_manager.cc:1888] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.758754 14259 log.cc:1196] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/fc32d7a811f741f4927b60b0bb9b6046
I20240501 19:56:34.759001 11106 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:34.759073 14259 ts_tablet_manager.cc:1909] T fc32d7a811f741f4927b60b0bb9b6046 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:34.759130 11106 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:34.759634 14261 ts_tablet_manager.cc:1888] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.759761 14261 log.cc:1196] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/fc32d7a811f741f4927b60b0bb9b6046
I20240501 19:56:34.759811 14259 tablet_replica.cc:330] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:34.759966 14259 raft_consensus.cc:2229] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.760107 14259 raft_consensus.cc:2258] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.760124 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet fc32d7a811f741f4927b60b0bb9b6046 (table rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]) successfully deleted
I20240501 19:56:34.760591 14259 ts_tablet_manager.cc:1875] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.761413 11106 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 86a125b33b0a4c109a27d7626ec4568e
I20240501 19:56:34.762109 11228 tablet_service.cc:1514] Processing DeleteTablet for tablet 86a125b33b0a4c109a27d7626ec4568e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:34.762171 13650 tablet_service.cc:1514] Processing DeleteTablet for tablet 86a125b33b0a4c109a27d7626ec4568e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:34.763240 14259 ts_tablet_manager.cc:1888] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.763267 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 86a125b33b0a4c109a27d7626ec4568e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:34 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:34.763312 14259 log.cc:1196] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/8fc636fe78c14563aac456322e1eda5c
I20240501 19:56:34.763558 14259 ts_tablet_manager.cc:1909] T 8fc636fe78c14563aac456322e1eda5c P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:34.764132 14259 tablet_replica.cc:330] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:34.764293 14259 raft_consensus.cc:2229] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.764305 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 8fc636fe78c14563aac456322e1eda5c (table rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]) successfully deleted
I20240501 19:56:34.764436 14261 ts_tablet_manager.cc:1909] T fc32d7a811f741f4927b60b0bb9b6046 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:34.764461 14259 raft_consensus.cc:2258] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.766216 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet fc32d7a811f741f4927b60b0bb9b6046 (table rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]) successfully deleted
I20240501 19:56:34.766297 14258 tablet_replica.cc:330] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:34.766408 14258 raft_consensus.cc:2229] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:34.766579 14258 raft_consensus.cc:2258] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.767113 14258 ts_tablet_manager.cc:1875] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.769376 14258 ts_tablet_manager.cc:1888] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:34.769513 14258 log.cc:1196] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/86a125b33b0a4c109a27d7626ec4568e
I20240501 19:56:34.769814 14258 ts_tablet_manager.cc:1909] T 86a125b33b0a4c109a27d7626ec4568e P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:34.770473 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 86a125b33b0a4c109a27d7626ec4568e (table rebalancer_dynamic_table_new_name [id=1407a4f230fa45469e0f6168d7549f59]) successfully deleted
I20240501 19:56:34.771206 14622 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.771528 14622 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:34.771695 14622 ts_tablet_manager.cc:1367] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.096s user 0.024s sys 0.001s
I20240501 19:56:34.772158 14622 raft_consensus.cc:348] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.772272 14622 raft_consensus.cc:729] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:34.772367 14622 consensus_queue.cc:260] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.772603 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet ea873ec8ced340b4ac082417c9b09aae from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:41129)
I20240501 19:56:34.772715 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet ea873ec8ced340b4ac082417c9b09aae from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:41129: session id = 6921cf2e41f64d66842885512da8de6c-ea873ec8ced340b4ac082417c9b09aae
I20240501 19:56:34.773238 14622 ts_tablet_manager.cc:1398] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:34.773633 11533 tablet_copy_source_session.cc:205] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.773842 14717 tablet_replica.cc:330] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:34.774262 11269 tablet_copy_service.cc:342] P 10ee60ead7b94c93ab749ec16a04b0dc: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-1c07d9fc05254a6399afcc54b3efb8ac received from {username='slave'} at 127.5.40.5:37827
I20240501 19:56:34.774333 11269 tablet_copy_service.cc:433] P 10ee60ead7b94c93ab749ec16a04b0dc: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-1c07d9fc05254a6399afcc54b3efb8ac on tablet 1c07d9fc05254a6399afcc54b3efb8ac with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:34.774148 14717 raft_consensus.cc:2229] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.774699 14717 raft_consensus.cc:2258] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.775648 14598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea873ec8ced340b4ac082417c9b09aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.776162 14717 ts_tablet_manager.cc:1875] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.776454 14259 ts_tablet_manager.cc:1875] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.777385 14598 tablet_copy_client.cc:783] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.777529 14598 tablet_copy_client.cc:647] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.778249 14717 ts_tablet_manager.cc:1888] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:34.778445 14259 ts_tablet_manager.cc:1888] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:34.778522 14717 log.cc:1196] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/86a125b33b0a4c109a27d7626ec4568e
I20240501 19:56:34.778856 14717 ts_tablet_manager.cc:1909] T 86a125b33b0a4c109a27d7626ec4568e P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting consensus metadata
I20240501 19:56:34.779186 14711 tablet_replica.cc:330] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:34.779359 14711 raft_consensus.cc:2229] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:34.779687 14711 raft_consensus.cc:2258] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.780086 14598 tablet_copy_client.cc:523] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.780478 14711 ts_tablet_manager.cc:1875] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.781229 14598 tablet_bootstrap.cc:492] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:34.781559 14259 log.cc:1196] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/86a125b33b0a4c109a27d7626ec4568e
I20240501 19:56:34.781895 14259 ts_tablet_manager.cc:1909] T 86a125b33b0a4c109a27d7626ec4568e P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:34.782968 14711 ts_tablet_manager.cc:1888] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.783088 14711 log.cc:1196] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/9e327331ed1a4790af392bf83e6a7cd2
I20240501 19:56:34.783354 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 86a125b33b0a4c109a27d7626ec4568e (table rebalancer_dynamic_table_new_name [id=1407a4f230fa45469e0f6168d7549f59]) successfully deleted
I20240501 19:56:34.783418 14711 ts_tablet_manager.cc:1909] T 9e327331ed1a4790af392bf83e6a7cd2 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:34.784080 14711 tablet_replica.cc:330] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:34.784159 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 9e327331ed1a4790af392bf83e6a7cd2 (table rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]) successfully deleted
I20240501 19:56:34.788347 14639 ts_tablet_manager.cc:902] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:34.788789 14639 tablet_copy_client.cc:308] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:34.784767 14622 ts_tablet_manager.cc:902] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:34.789435 14622 tablet_copy_client.cc:308] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:34.793121 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet b2ecce0ca7a941dbb98b89b797beb8a1 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:44731)
I20240501 19:56:34.793221 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet b2ecce0ca7a941dbb98b89b797beb8a1 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:44731: session id = 533908e517b24f77b2af8b5863e4ae5a-b2ecce0ca7a941dbb98b89b797beb8a1
I20240501 19:56:34.794139 11664 tablet_copy_source_session.cc:205] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.794907 14639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ecce0ca7a941dbb98b89b797beb8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.790314 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 5e63472cfff449089d88d18b9e495912 from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:53821)
I20240501 19:56:34.795600 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 5e63472cfff449089d88d18b9e495912 from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:53821: session id = 8125d8ad6d0d44caae9c5c90c90df96b-5e63472cfff449089d88d18b9e495912
I20240501 19:56:34.795629 14711 raft_consensus.cc:2229] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.795779 14711 raft_consensus.cc:2258] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.796272 14711 ts_tablet_manager.cc:1875] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:34.796573 11533 tablet_copy_source_session.cc:205] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.796983 14639 tablet_copy_client.cc:783] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.797152 14639 tablet_copy_client.cc:647] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.798034 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e63472cfff449089d88d18b9e495912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.798688 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet 86a125b33b0a4c109a27d7626ec4568e (table rebalancer_dynamic_table_new_name [id=1407a4f230fa45469e0f6168d7549f59]) successfully deleted
I20240501 19:56:34.798837 14639 tablet_copy_client.cc:523] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.800087 14639 tablet_bootstrap.cc:492] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:34.800480 14622 tablet_copy_client.cc:783] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.800626 14622 tablet_copy_client.cc:647] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.801764 14622 tablet_copy_client.cc:523] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.803097 14622 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:34.807636 14711 ts_tablet_manager.cc:1888] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:34.807734 14711 log.cc:1196] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/8fc636fe78c14563aac456322e1eda5c
I20240501 19:56:34.808050 14711 ts_tablet_manager.cc:1909] T 8fc636fe78c14563aac456322e1eda5c P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:34.808697 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 8fc636fe78c14563aac456322e1eda5c (table rebalance_test_table_4 [id=1a5caf92dc994323b0c734064329dce1]) successfully deleted
I20240501 19:56:34.822264 14718 ts_tablet_manager.cc:902] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:34.822422 14639 tablet_bootstrap.cc:492] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.822695 14639 tablet_bootstrap.cc:492] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:34.822878 14639 ts_tablet_manager.cc:1367] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.000s
I20240501 19:56:34.822863 14718 tablet_copy_client.cc:308] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:34.823184 14734 ts_tablet_manager.cc:902] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:34.823530 11401 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet 4f1f6382bcc84fcda444dcfae87b00c1 from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:36785)
I20240501 19:56:34.823664 11401 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet 4f1f6382bcc84fcda444dcfae87b00c1 from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:36785: session id = 8125d8ad6d0d44caae9c5c90c90df96b-4f1f6382bcc84fcda444dcfae87b00c1
I20240501 19:56:34.824039 14734 tablet_copy_client.cc:308] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:34.825119 11401 tablet_copy_source_session.cc:205] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.826217 14718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1f6382bcc84fcda444dcfae87b00c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.828855 14718 tablet_copy_client.cc:783] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.829057 14718 tablet_copy_client.cc:647] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.829871 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet e307d45fb57f444ca09d94d59b9bcd64 from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:54335)
I20240501 19:56:34.829990 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet e307d45fb57f444ca09d94d59b9bcd64 from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:54335: session id = 8125d8ad6d0d44caae9c5c90c90df96b-e307d45fb57f444ca09d94d59b9bcd64
I20240501 19:56:34.830406 14639 raft_consensus.cc:348] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.830541 14639 raft_consensus.cc:729] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:34.830641 14639 consensus_queue.cc:260] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.830971 11664 tablet_copy_source_session.cc:205] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.831390 14639 ts_tablet_manager.cc:1398] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240501 19:56:34.832266 14622 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.832509 14622 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:34.832655 14622 ts_tablet_manager.cc:1367] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.002s
I20240501 19:56:34.832717 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-b2ecce0ca7a941dbb98b89b797beb8a1 received from {username='slave'} at 127.5.40.8:44731
I20240501 19:56:34.832783 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-b2ecce0ca7a941dbb98b89b797beb8a1 on tablet b2ecce0ca7a941dbb98b89b797beb8a1 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:34.833031 14622 raft_consensus.cc:348] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.833134 14622 raft_consensus.cc:729] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:34.833215 14622 consensus_queue.cc:260] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.833261 14734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e307d45fb57f444ca09d94d59b9bcd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.833467 14622 ts_tablet_manager.cc:1398] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:34.835408 14734 tablet_copy_client.cc:783] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.835574 14734 tablet_copy_client.cc:647] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.835986 14598 tablet_bootstrap.cc:492] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.836222 14598 tablet_bootstrap.cc:492] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:34.836360 14598 ts_tablet_manager.cc:1367] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.055s user 0.025s sys 0.000s
I20240501 19:56:34.836753 14598 raft_consensus.cc:348] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:34.836853 14598 raft_consensus.cc:729] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:34.836946 14598 consensus_queue.cc:260] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:34.837675 14598 ts_tablet_manager.cc:1398] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:34.838286 11532 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-ea873ec8ced340b4ac082417c9b09aae received from {username='slave'} at 127.5.40.7:41129
I20240501 19:56:34.838361 11532 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-ea873ec8ced340b4ac082417c9b09aae on tablet ea873ec8ced340b4ac082417c9b09aae with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:34.839515 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-5e63472cfff449089d88d18b9e495912 received from {username='slave'} at 127.5.40.5:53821
I20240501 19:56:34.839579 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-5e63472cfff449089d88d18b9e495912 on tablet 5e63472cfff449089d88d18b9e495912 with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:34.841292 14603 ts_tablet_manager.cc:902] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:34.841789 14603 tablet_copy_client.cc:308] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:34.842377 14672 ts_tablet_manager.cc:902] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:34.842751 14672 tablet_copy_client.cc:308] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:34.843847 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet 582d72e630c84088a6e4a7309ae7e952 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:37783)
I20240501 19:56:34.843945 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet 582d72e630c84088a6e4a7309ae7e952 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:37783: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-582d72e630c84088a6e4a7309ae7e952
I20240501 19:56:34.844861 11664 tablet_copy_source_session.cc:205] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.845697 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582d72e630c84088a6e4a7309ae7e952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.845692 11401 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet 70fcac1c11404f51838dcb000f76d737 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:39229)
I20240501 19:56:34.845789 11401 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet 70fcac1c11404f51838dcb000f76d737 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:39229: session id = 533908e517b24f77b2af8b5863e4ae5a-70fcac1c11404f51838dcb000f76d737
I20240501 19:56:34.846642 11401 tablet_copy_source_session.cc:205] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.847762 14672 tablet_copy_client.cc:783] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.847888 14672 tablet_copy_client.cc:647] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.848575 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fcac1c11404f51838dcb000f76d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.850428 14603 tablet_copy_client.cc:783] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.850554 14603 tablet_copy_client.cc:647] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.850683 14672 tablet_copy_client.cc:523] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.851850 14672 tablet_bootstrap.cc:492] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:34.851971 14718 tablet_copy_client.cc:523] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.853263 14718 tablet_bootstrap.cc:492] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:34.856662 14603 tablet_copy_client.cc:523] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.857879 14734 tablet_copy_client.cc:523] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.857905 14603 tablet_bootstrap.cc:492] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:34.859098 14734 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:34.863101 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:34.868867 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 9850f9a07d9344a9b39ada629073aee8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e]), partition=
I20240501 19:56:34.868984 13911 tablet_service.cc:1467] Processing CreateTablet for tablet 9850f9a07d9344a9b39ada629073aee8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e]), partition=
I20240501 19:56:34.869119 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850f9a07d9344a9b39ada629073aee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.869181 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850f9a07d9344a9b39ada629073aee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.871308 14580 tablet_bootstrap.cc:492] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:34.872220 14580 tablet_bootstrap.cc:654] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.872790 14587 tablet_bootstrap.cc:492] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:34.872922 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 9850f9a07d9344a9b39ada629073aee8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e]), partition=
I20240501 19:56:34.873180 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850f9a07d9344a9b39ada629073aee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.873566 14587 tablet_bootstrap.cc:654] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.875242 14672 tablet_bootstrap.cc:492] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.875488 14672 tablet_bootstrap.cc:492] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:34.875635 14672 ts_tablet_manager.cc:1367] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.008s
I20240501 19:56:34.876226 14672 raft_consensus.cc:348] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.876341 14672 raft_consensus.cc:729] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:34.876433 14672 consensus_queue.cc:260] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.876686 14672 ts_tablet_manager.cc:1398] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:34.877382 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-582d72e630c84088a6e4a7309ae7e952 received from {username='slave'} at 127.5.40.6:37783
I20240501 19:56:34.877465 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-582d72e630c84088a6e4a7309ae7e952 on tablet 582d72e630c84088a6e4a7309ae7e952 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:34.883232 14753 tablet_bootstrap.cc:492] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:34.884135 14753 tablet_bootstrap.cc:654] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.886298 14580 tablet_bootstrap.cc:492] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:34.886394 14580 ts_tablet_manager.cc:1367] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:56:34.886875 14580 raft_consensus.cc:348] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:34.886987 14580 raft_consensus.cc:374] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.887022 14580 raft_consensus.cc:729] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.887141 14580 consensus_queue.cc:260] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:34.887423 14580 ts_tablet_manager.cc:1398] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.888350 14587 tablet_bootstrap.cc:492] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:34.888466 14587 ts_tablet_manager.cc:1367] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:56:34.888933 14587 raft_consensus.cc:348] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:34.889043 14587 raft_consensus.cc:374] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.889073 14587 raft_consensus.cc:729] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.889191 14587 consensus_queue.cc:260] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:34.889448 14587 ts_tablet_manager.cc:1398] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:34.890702 14753 tablet_bootstrap.cc:492] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:34.890801 14753 ts_tablet_manager.cc:1367] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:34.891253 14753 raft_consensus.cc:348] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:34.893885 14753 raft_consensus.cc:374] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.893968 14639 ts_tablet_manager.cc:902] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:34.893968 14753 raft_consensus.cc:729] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.894181 14753 consensus_queue.cc:260] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:34.894486 14639 tablet_copy_client.cc:308] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:34.894536 14753 ts_tablet_manager.cc:1398] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:34.897514 14718 tablet_bootstrap.cc:492] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.897830 14718 tablet_bootstrap.cc:492] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:34.898008 14718 ts_tablet_manager.cc:1367] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.045s user 0.024s sys 0.004s
I20240501 19:56:34.898466 14718 raft_consensus.cc:348] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.898602 14718 raft_consensus.cc:729] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:34.898847 14718 consensus_queue.cc:260] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.900938 14718 ts_tablet_manager.cc:1398] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:34.902348 11401 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet cebd15701ee44b11a2f8804740fb4299 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:39229)
I20240501 19:56:34.902482 11401 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet cebd15701ee44b11a2f8804740fb4299 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:39229: session id = 533908e517b24f77b2af8b5863e4ae5a-cebd15701ee44b11a2f8804740fb4299
I20240501 19:56:34.903414 11401 tablet_copy_source_session.cc:205] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.904331 14639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cebd15701ee44b11a2f8804740fb4299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.904670 14734 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.904933 14734 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:34.905102 14734 ts_tablet_manager.cc:1367] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.046s user 0.018s sys 0.004s
I20240501 19:56:34.905512 14734 raft_consensus.cc:348] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.905630 14734 raft_consensus.cc:729] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:34.905725 14734 consensus_queue.cc:260] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:34.906163 14734 ts_tablet_manager.cc:1398] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:34.906483 14639 tablet_copy_client.cc:783] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.906633 14639 tablet_copy_client.cc:647] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.907100 14603 tablet_bootstrap.cc:492] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.907347 14603 tablet_bootstrap.cc:492] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:34.907532 14603 ts_tablet_manager.cc:1367] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.050s user 0.025s sys 0.001s
I20240501 19:56:34.907522 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-e307d45fb57f444ca09d94d59b9bcd64 received from {username='slave'} at 127.5.40.5:54335
I20240501 19:56:34.907595 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-e307d45fb57f444ca09d94d59b9bcd64 on tablet e307d45fb57f444ca09d94d59b9bcd64 with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:34.907981 14603 raft_consensus.cc:348] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.908103 14603 raft_consensus.cc:729] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:34.908207 14603 consensus_queue.cc:260] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.909076 14603 ts_tablet_manager.cc:1398] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:34.909579 11401 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-70fcac1c11404f51838dcb000f76d737 received from {username='slave'} at 127.5.40.8:39229
I20240501 19:56:34.909675 11401 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-70fcac1c11404f51838dcb000f76d737 on tablet 70fcac1c11404f51838dcb000f76d737 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:34.909967 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-4f1f6382bcc84fcda444dcfae87b00c1 received from {username='slave'} at 127.5.40.5:36785
I20240501 19:56:34.910040 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-4f1f6382bcc84fcda444dcfae87b00c1 on tablet 4f1f6382bcc84fcda444dcfae87b00c1 with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:34.910614 14639 tablet_copy_client.cc:523] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.911888 14639 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:34.919271 14618 raft_consensus.cc:482] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:34.919371 14618 raft_consensus.cc:504] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.919835 14618 leader_election.cc:288] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:34.920457 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96511455168148b7b619670f635d3a3e" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:34.920586 11249 raft_consensus.cc:2454] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:34.920874 13603 leader_election.cc:302] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:34.921056 14618 raft_consensus.cc:2781] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:34.921118 14618 raft_consensus.cc:482] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:34.921151 14618 raft_consensus.cc:3037] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.922148 14618 raft_consensus.cc:504] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.922482 14618 leader_election.cc:288] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:34.923658 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96511455168148b7b619670f635d3a3e" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:34.923775 11249 raft_consensus.cc:3037] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.924511 14672 ts_tablet_manager.cc:902] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:34.924734 11249 raft_consensus.cc:2454] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:34.924911 14672 tablet_copy_client.cc:308] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:34.925071 13603 leader_election.cc:302] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:34.925645 14618 raft_consensus.cc:2781] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:34.925745 14618 raft_consensus.cc:686] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:34.925889 14618 consensus_queue.cc:237] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:34.927549 13932 raft_consensus.cc:1203] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:34.928289 14063 raft_consensus.cc:1203] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:34.933892 11105 catalog_manager.cc:5484] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.934669 14639 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.934953 14639 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:34.935151 14639 ts_tablet_manager.cc:1367] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.003s
I20240501 19:56:34.935657 14639 raft_consensus.cc:348] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.935776 14639 raft_consensus.cc:729] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:34.935874 14639 consensus_queue.cc:260] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:34.936067 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96511455168148b7b619670f635d3a3e" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:34.936156 14639 ts_tablet_manager.cc:1398] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:34.936172 14062 raft_consensus.cc:3037] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.936587 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 04932055883c412f944e839982815ab6 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:51789)
I20240501 19:56:34.936690 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 04932055883c412f944e839982815ab6 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:51789: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-04932055883c412f944e839982815ab6
I20240501 19:56:34.937152 14062 raft_consensus.cc:2454] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:34.937616 11533 tablet_copy_source_session.cc:205] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.937827 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-cebd15701ee44b11a2f8804740fb4299 received from {username='slave'} at 127.5.40.8:39229
I20240501 19:56:34.937916 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-cebd15701ee44b11a2f8804740fb4299 on tablet cebd15701ee44b11a2f8804740fb4299 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:34.936076 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96511455168148b7b619670f635d3a3e" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:34.938128 14063 raft_consensus.cc:2362] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1. Re-sending same reply.
I20240501 19:56:34.939560 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04932055883c412f944e839982815ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.941454 14672 tablet_copy_client.cc:783] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.941637 14672 tablet_copy_client.cc:647] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.944939 14672 tablet_copy_client.cc:523] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:34.946305 14672 tablet_bootstrap.cc:492] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:34.952193 11624 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:34.952881 11491 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 19:56:34.965399 14043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:34.966346 13911 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:34.967089 13781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:34.973414 14598 ts_tablet_manager.cc:902] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:34.975723 14598 tablet_copy_client.cc:308] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:34.976428 14672 tablet_bootstrap.cc:492] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:34.976711 14672 tablet_bootstrap.cc:492] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:34.976895 14672 ts_tablet_manager.cc:1367] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.000s
I20240501 19:56:34.977531 14672 raft_consensus.cc:348] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.977794 14672 raft_consensus.cc:729] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:34.977912 14672 consensus_queue.cc:260] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:34.978292 14672 ts_tablet_manager.cc:1398] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:34.978751 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-04932055883c412f944e839982815ab6 received from {username='slave'} at 127.5.40.6:51789
I20240501 19:56:34.978837 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-04932055883c412f944e839982815ab6 on tablet 04932055883c412f944e839982815ab6 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:34.980393 11228 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:34.986668 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet 69facf86bba24f209c06afafc1ba8f7b from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:33205)
I20240501 19:56:34.986791 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet 69facf86bba24f209c06afafc1ba8f7b from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:33205: session id = 6921cf2e41f64d66842885512da8de6c-69facf86bba24f209c06afafc1ba8f7b
I20240501 19:56:34.987851 11664 tablet_copy_source_session.cc:205] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:34.988770 14598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69facf86bba24f209c06afafc1ba8f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.990654 14598 tablet_copy_client.cc:783] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:34.990797 14598 tablet_copy_client.cc:647] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:34.994702 11359 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:35.007261 13650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:35.018987 14672 ts_tablet_manager.cc:902] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:35.019495 14672 tablet_copy_client.cc:308] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:35.019942 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet 3d99f4d3127348dba949ab5fd4666956 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:37783)
I20240501 19:56:35.020054 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet 3d99f4d3127348dba949ab5fd4666956 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:37783: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-3d99f4d3127348dba949ab5fd4666956
I20240501 19:56:35.021060 11664 tablet_copy_source_session.cc:205] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:35.021209 14598 tablet_copy_client.cc:523] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:35.022684 14598 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:35.027849 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d99f4d3127348dba949ab5fd4666956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.030112 14672 tablet_copy_client.cc:783] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:35.030262 14672 tablet_copy_client.cc:647] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:35.034476 14672 tablet_copy_client.cc:523] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:35.035898 14672 tablet_bootstrap.cc:492] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:35.059213 14598 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:35.059581 14598 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:35.059779 14598 ts_tablet_manager.cc:1367] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.037s user 0.023s sys 0.003s
I20240501 19:56:35.060348 14598 raft_consensus.cc:348] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:35.060493 14598 raft_consensus.cc:729] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:35.060602 14598 consensus_queue.cc:260] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:35.061825 14672 tablet_bootstrap.cc:492] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:35.062105 14672 tablet_bootstrap.cc:492] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:35.062279 14672 ts_tablet_manager.cc:1367] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.003s
I20240501 19:56:35.062744 14672 raft_consensus.cc:348] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:35.062865 14672 raft_consensus.cc:729] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:35.063014 14672 consensus_queue.cc:260] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:35.063114 14598 ts_tablet_manager.cc:1398] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:35.063308 14672 ts_tablet_manager.cc:1398] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:56:35.064021 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-3d99f4d3127348dba949ab5fd4666956 received from {username='slave'} at 127.5.40.6:37783
I20240501 19:56:35.064100 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-3d99f4d3127348dba949ab5fd4666956 on tablet 3d99f4d3127348dba949ab5fd4666956 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:35.066040 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-69facf86bba24f209c06afafc1ba8f7b received from {username='slave'} at 127.5.40.7:33205
I20240501 19:56:35.066112 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-69facf86bba24f209c06afafc1ba8f7b on tablet 69facf86bba24f209c06afafc1ba8f7b with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:35.073776 14439 raft_consensus.cc:482] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.073913 14439 raft_consensus.cc:504] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:35.074389 14439 leader_election.cc:288] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:35.074993 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850f9a07d9344a9b39ada629073aee8" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:35.075140 13932 raft_consensus.cc:2454] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:35.075691 13734 leader_election.cc:302] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:35.075937 14439 raft_consensus.cc:2781] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.076001 14439 raft_consensus.cc:482] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.076037 14439 raft_consensus.cc:3037] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.076356 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850f9a07d9344a9b39ada629073aee8" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:35.076490 11643 raft_consensus.cc:2454] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:35.077068 14439 raft_consensus.cc:504] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:35.077385 14439 leader_election.cc:288] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:35.078083 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850f9a07d9344a9b39ada629073aee8" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:35.078205 13932 raft_consensus.cc:3037] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.078450 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850f9a07d9344a9b39ada629073aee8" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:35.078541 11643 raft_consensus.cc:3037] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.079190 13932 raft_consensus.cc:2454] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:35.079460 11643 raft_consensus.cc:2454] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:35.079694 13734 leader_election.cc:302] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:35.079901 14439 raft_consensus.cc:2781] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.080052 14439 raft_consensus.cc:686] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:35.080356 14439 consensus_queue.cc:237] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:35.080899 14618 consensus_queue.cc:1035] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.083145 11105 catalog_manager.cc:5484] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.086388 14618 consensus_queue.cc:1035] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.086937 11511 consensus_queue.cc:237] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:35.089514 11643 raft_consensus.cc:1261] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:35.090307 11249 raft_consensus.cc:1261] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:35.090718 13437 consensus_queue.cc:1035] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:35.090883 11445 consensus_peers.cc:485] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85ba57eb6aae4f12b4a4641d633104a0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:35.091061 13437 consensus_queue.cc:1035] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:35.092743 14391 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:35.093236 11249 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:35.095243 11106 catalog_manager.cc:5484] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.096377 11643 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:35.104497 11511 consensus_queue.cc:237] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:35.105872 11249 raft_consensus.cc:1261] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:35.105934 11377 raft_consensus.cc:1261] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:35.105953 11448 consensus_peers.cc:485] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dc0294d0dc5427e80411be5db9c95c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:35.106333 13437 consensus_queue.cc:1035] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:35.106530 14391 consensus_queue.cc:1035] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:35.108247 13437 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:35.109100 11377 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:35.109211 11249 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:35.111845 11106 catalog_manager.cc:5484] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.120831 11511 consensus_queue.cc:237] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:35.122164 11643 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:35.122193 11249 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:35.122300 11445 consensus_peers.cc:485] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7073666bc3f4c0a8c1e003fb1ef4fef. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:35.122664 13437 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:35.122849 14391 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:35.124518 14391 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:35.125125 11249 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:35.126525 11643 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:35.127923 11105 catalog_manager.cc:5484] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.177476 14063 raft_consensus.cc:1203] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.180519 13800 raft_consensus.cc:3037] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.182046 14797 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.191331 13670 raft_consensus.cc:1203] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.194828 14063 raft_consensus.cc:1203] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.200083 13801 raft_consensus.cc:1203] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.202683 14280 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.210131 13670 raft_consensus.cc:1203] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.228678 11099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:35.234917 13932 raft_consensus.cc:1261] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.235358 14439 consensus_queue.cc:1035] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.236277 11643 raft_consensus.cc:1261] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.237205 14439 consensus_queue.cc:1035] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.238250 11106 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet 9850f9a07d9344a9b39ada629073aee8 (table rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e]). Expected version 1 got 0
I20240501 19:56:35.239367 14875 mvcc.cc:205] Tried to move back new op lower bound from 7022974546876542976 to 7022974546252611584. Current Snapshot: MvccSnapshot[applied={T|T < 7022974546876542976}]
I20240501 19:56:35.240017 11106 catalog_manager.cc:5576] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331) does not have the latest schema for tablet 9850f9a07d9344a9b39ada629073aee8 (table rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e]). Expected version 1 got 0
I20240501 19:56:35.240955 11106 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet 9850f9a07d9344a9b39ada629073aee8 (table rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e]). Expected version 1 got 0
I20240501 19:56:35.241379 11105 catalog_manager.cc:5576] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041) does not have the latest schema for tablet 9850f9a07d9344a9b39ada629073aee8 (table rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e]). Expected version 1 got 0
I20240501 19:56:35.241586 14896 tablet.cc:1722] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:35.242086 14896 tablet.cc:1620] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:35.243680 11105 catalog_manager.cc:5576] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331) does not have the latest schema for tablet 9850f9a07d9344a9b39ada629073aee8 (table rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e]). Expected version 1 got 0
I20240501 19:56:35.244674 11105 catalog_manager.cc:5576] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041) does not have the latest schema for tablet 9850f9a07d9344a9b39ada629073aee8 (table rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e]). Expected version 1 got 0
I20240501 19:56:35.245739 11105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 19:56:35.245893 11105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 19:56:35.249454 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e] alter complete (version 1)
I20240501 19:56:35.249508 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 1c43596467ca48a58c42ff487b3fce90
I20240501 19:56:35.250046 11228 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c43596467ca48a58c42ff487b3fce90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:35.250211 14259 tablet_replica.cc:330] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:35.250344 14259 raft_consensus.cc:2229] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:35.250523 14259 raft_consensus.cc:2258] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.250514 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c43596467ca48a58c42ff487b3fce90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:35.250770 14261 tablet_replica.cc:330] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:35.250906 14261 raft_consensus.cc:2229] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.250975 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet df804ea6956f4c15a8990e3a12945ff8
I20240501 19:56:35.251075 14261 raft_consensus.cc:2258] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.251067 14259 ts_tablet_manager.cc:1875] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.251655 14261 ts_tablet_manager.cc:1875] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.251783 14890 tablet.cc:1722] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:35.251902 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet df804ea6956f4c15a8990e3a12945ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:35.252476 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 6f2e2992240f49bdb3e81e5516199277
I20240501 19:56:35.252617 14890 tablet.cc:1620] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:35.252902 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f2e2992240f49bdb3e81e5516199277 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:35.253104 14898 tablet.cc:1722] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:35.253293 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f2e2992240f49bdb3e81e5516199277 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:35.253407 14898 tablet.cc:1620] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: MemRowSet was empty: no flush needed.
I20240501 19:56:35.253502 14259 ts_tablet_manager.cc:1888] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.253566 14259 log.cc:1196] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c43596467ca48a58c42ff487b3fce90
I20240501 19:56:35.253926 14261 ts_tablet_manager.cc:1888] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.253991 14261 log.cc:1196] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c43596467ca48a58c42ff487b3fce90
I20240501 19:56:35.253984 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c43596467ca48a58c42ff487b3fce90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:35.254125 14258 tablet_replica.cc:330] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:35.254220 14261 ts_tablet_manager.cc:1909] T 1c43596467ca48a58c42ff487b3fce90 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:35.254243 14258 raft_consensus.cc:2229] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.254369 14258 raft_consensus.cc:2258] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.254706 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet df804ea6956f4c15a8990e3a12945ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:35.254953 14258 ts_tablet_manager.cc:1875] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.254974 14711 tablet_replica.cc:330] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:35.255122 14711 raft_consensus.cc:2229] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:35.255338 14711 raft_consensus.cc:2258] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.255870 14711 ts_tablet_manager.cc:1875] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.257007 14258 ts_tablet_manager.cc:1888] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.257066 14258 log.cc:1196] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/df804ea6956f4c15a8990e3a12945ff8
I20240501 19:56:35.257288 14258 ts_tablet_manager.cc:1909] T df804ea6956f4c15a8990e3a12945ff8 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:35.257617 14711 ts_tablet_manager.cc:1888] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.257680 14711 log.cc:1196] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/df804ea6956f4c15a8990e3a12945ff8
I20240501 19:56:35.257776 11228 tablet_service.cc:1514] Processing DeleteTablet for tablet df804ea6956f4c15a8990e3a12945ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:35.257921 14711 ts_tablet_manager.cc:1909] T df804ea6956f4c15a8990e3a12945ff8 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:35.257977 14258 tablet_replica.cc:330] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:35.258090 14258 raft_consensus.cc:2229] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.258203 14258 raft_consensus.cc:2258] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.253818 14259 ts_tablet_manager.cc:1909] T 1c43596467ca48a58c42ff487b3fce90 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:35.258481 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 1c43596467ca48a58c42ff487b3fce90 (table rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]) successfully deleted
I20240501 19:56:35.258639 14258 ts_tablet_manager.cc:1875] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.258689 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f2e2992240f49bdb3e81e5516199277 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:35.258950 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet df804ea6956f4c15a8990e3a12945ff8 (table rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]) successfully deleted
I20240501 19:56:35.260008 14259 tablet_replica.cc:330] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:35.260226 14259 raft_consensus.cc:2229] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:35.260345 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 1c43596467ca48a58c42ff487b3fce90 (table rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]) successfully deleted
I20240501 19:56:35.260499 14258 ts_tablet_manager.cc:1888] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.260627 14258 log.cc:1196] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f2e2992240f49bdb3e81e5516199277
I20240501 19:56:35.260802 14259 raft_consensus.cc:2258] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.261013 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet df804ea6956f4c15a8990e3a12945ff8 (table rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]) successfully deleted
I20240501 19:56:35.261210 14261 tablet_replica.cc:330] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:35.261245 14259 ts_tablet_manager.cc:1875] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.261530 14261 raft_consensus.cc:2229] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.261669 14258 ts_tablet_manager.cc:1909] T 6f2e2992240f49bdb3e81e5516199277 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:35.261726 14261 raft_consensus.cc:2258] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.262190 14258 tablet_replica.cc:330] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:35.262312 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 6f2e2992240f49bdb3e81e5516199277 (table rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]) successfully deleted
I20240501 19:56:35.262409 14258 raft_consensus.cc:2229] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.262528 14258 raft_consensus.cc:2258] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.262609 14261 ts_tablet_manager.cc:1875] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.264415 14259 ts_tablet_manager.cc:1888] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.264537 14259 log.cc:1196] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f2e2992240f49bdb3e81e5516199277
I20240501 19:56:35.264782 14259 ts_tablet_manager.cc:1909] T 6f2e2992240f49bdb3e81e5516199277 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:35.264779 14258 ts_tablet_manager.cc:1875] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.265291 14259 tablet_replica.cc:330] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:35.265393 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 6f2e2992240f49bdb3e81e5516199277 (table rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]) successfully deleted
I20240501 19:56:35.265419 14259 raft_consensus.cc:2229] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.265542 14259 raft_consensus.cc:2258] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.265920 14259 ts_tablet_manager.cc:1875] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.266068 14261 ts_tablet_manager.cc:1888] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.266144 14261 log.cc:1196] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f2e2992240f49bdb3e81e5516199277
I20240501 19:56:35.266413 14261 ts_tablet_manager.cc:1909] T 6f2e2992240f49bdb3e81e5516199277 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:35.268831 14259 ts_tablet_manager.cc:1888] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.269098 14259 log.cc:1196] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/df804ea6956f4c15a8990e3a12945ff8
I20240501 19:56:35.269572 14259 ts_tablet_manager.cc:1909] T df804ea6956f4c15a8990e3a12945ff8 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:35.270992 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet df804ea6956f4c15a8990e3a12945ff8 (table rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]) successfully deleted
I20240501 19:56:35.272130 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 6f2e2992240f49bdb3e81e5516199277 (table rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]) successfully deleted
I20240501 19:56:35.272693 14258 ts_tablet_manager.cc:1888] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.272759 14258 log.cc:1196] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/1c43596467ca48a58c42ff487b3fce90
I20240501 19:56:35.272958 14258 ts_tablet_manager.cc:1909] T 1c43596467ca48a58c42ff487b3fce90 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:35.273583 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 1c43596467ca48a58c42ff487b3fce90 (table rebalance_test_table_6 [id=28cd254c5f624c2e9a2b247aef5b8e36]) successfully deleted
I20240501 19:56:35.281946 13670 raft_consensus.cc:1203] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.284384 13670 raft_consensus.cc:1203] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.289126 13801 raft_consensus.cc:1203] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.313609 13801 raft_consensus.cc:1203] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.358098 14280 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.358564 13932 raft_consensus.cc:1203] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.359655 14062 raft_consensus.cc:1203] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.360868 11099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:35.364935 14797 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.368271 14890 tablet.cc:1722] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:35.368613 14890 tablet.cc:1620] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:35.369477 14896 tablet.cc:1722] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:35.369776 14896 tablet.cc:1620] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:35.371212 14898 tablet.cc:1722] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:35.371529 14898 tablet.cc:1620] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: MemRowSet was empty: no flush needed.
I20240501 19:56:35.374477 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=ef1f8bc1aa384faa83a6f7224939cd4e] alter complete (version 2)
I20240501 19:56:35.425467 14063 raft_consensus.cc:1203] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.425467 13670 raft_consensus.cc:1203] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.476670 11099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:35.482268 14890 tablet.cc:1722] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:35.482458 14896 tablet.cc:1722] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:35.485270 14898 tablet.cc:1722] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:35.486727 11099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=ef1f8bc1aa384faa83a6f7224939cd4e] alter complete (version 3)
I20240501 19:56:35.490661 13932 raft_consensus.cc:1203] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.492542 11105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:35.493635 14199 raft_consensus.cc:1050] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:35.494112 14199 consensus_queue.cc:237] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.495836 11511 raft_consensus.cc:1261] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.495959 14598 ts_tablet_manager.cc:902] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:35.495972 14063 raft_consensus.cc:1261] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.496399 14598 tablet_copy_client.cc:308] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:35.496577 11377 raft_consensus.cc:1261] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.496840 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 85ba57eb6aae4f12b4a4641d633104a0 from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:41129)
I20240501 19:56:35.496946 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 85ba57eb6aae4f12b4a4641d633104a0 from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:41129: session id = 6921cf2e41f64d66842885512da8de6c-85ba57eb6aae4f12b4a4641d633104a0
I20240501 19:56:35.497001 14356 consensus_queue.cc:1035] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.497378 14356 consensus_queue.cc:1035] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.497660 14356 raft_consensus.cc:1050] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:35.497982 11533 tablet_copy_source_session.cc:205] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:35.497997 14356 consensus_queue.cc:237] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:35.499043 14937 consensus_queue.cc:1035] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.500772 11377 raft_consensus.cc:1261] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.501086 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3883b8a45041328ea220184db9f8ce (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.501134 14199 consensus_queue.cc:1035] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.501134 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 8078e0cbffa943c98584224d05b0ece0 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.501394 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3883b8a45041328ea220184db9f8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.501596 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8078e0cbffa943c98584224d05b0ece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.501969 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 8078e0cbffa943c98584224d05b0ece0 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.501989 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc692e320d94825964040105a3175cd (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.502221 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc692e320d94825964040105a3175cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.502220 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8078e0cbffa943c98584224d05b0ece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.503782 14947 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:35.504514 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc692e320d94825964040105a3175cd (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.504717 14947 tablet_bootstrap.cc:654] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.504732 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc692e320d94825964040105a3175cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.505296 14598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ba57eb6aae4f12b4a4641d633104a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.505820 14947 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:35.505893 14947 ts_tablet_manager.cc:1367] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:35.506338 14947 raft_consensus.cc:348] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:35.506435 14947 raft_consensus.cc:374] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.506464 14947 raft_consensus.cc:729] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.506592 14947 consensus_queue.cc:260] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:35.506623 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3883b8a45041328ea220184db9f8ce (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.506851 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3883b8a45041328ea220184db9f8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.506873 14947 ts_tablet_manager.cc:1398] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:35.507048 14947 tablet_bootstrap.cc:492] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:35.507359 14598 tablet_copy_client.cc:783] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:35.508013 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3883b8a45041328ea220184db9f8ce (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.508289 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3883b8a45041328ea220184db9f8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.509248 11643 raft_consensus.cc:1261] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.509932 13649 tablet_service.cc:1467] Processing CreateTablet for tablet 8078e0cbffa943c98584224d05b0ece0 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.510010 14949 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:35.510049 14943 consensus_queue.cc:1035] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.510154 13649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8078e0cbffa943c98584224d05b0ece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.511011 13932 raft_consensus.cc:1261] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.511843 14949 tablet_bootstrap.cc:654] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.512193 14598 tablet_copy_client.cc:647] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:35.512243 13648 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc692e320d94825964040105a3175cd (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:35.512811 14199 consensus_queue.cc:1035] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.512447 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc692e320d94825964040105a3175cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.514119 14598 tablet_copy_client.cc:523] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:35.513995 14199 raft_consensus.cc:2932] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.514235 14952 tablet_bootstrap.cc:492] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:35.515107 14952 tablet_bootstrap.cc:654] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.515431 14598 tablet_bootstrap.cc:492] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:35.515802 11377 raft_consensus.cc:2932] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.516413 14063 raft_consensus.cc:2932] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.517150 14947 tablet_bootstrap.cc:654] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.517851 14954 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:35.518662 14954 tablet_bootstrap.cc:654] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.519954 11105 catalog_manager.cc:5484] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.521584 14937 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.522059 11643 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.522222 14948 tablet_bootstrap.cc:492] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:35.523116 14948 tablet_bootstrap.cc:654] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.524544 11377 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.525920 11100 catalog_manager.cc:5484] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.526631 11249 consensus_queue.cc:237] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.528682 11378 raft_consensus.cc:1261] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.529237 14063 raft_consensus.cc:1261] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.529572 14199 consensus_queue.cc:1035] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.529812 14199 consensus_queue.cc:1035] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.530462 14947 tablet_bootstrap.cc:492] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:35.530562 14947 ts_tablet_manager.cc:1367] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.002s
I20240501 19:56:35.531057 14947 raft_consensus.cc:348] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.531185 14947 raft_consensus.cc:374] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.531222 14947 raft_consensus.cc:729] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.531342 14947 consensus_queue.cc:260] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.531634 14947 ts_tablet_manager.cc:1398] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:35.532766 13932 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.533272 14937 raft_consensus.cc:2932] T c92c6475f5ed4529b1efd12b6d495a11 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.534015 11378 raft_consensus.cc:2932] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.534158 14954 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:35.534262 14954 ts_tablet_manager.cc:1367] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240501 19:56:35.534723 14954 raft_consensus.cc:348] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:35.534828 14954 raft_consensus.cc:374] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.534862 14954 raft_consensus.cc:729] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.534993 14954 consensus_queue.cc:260] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:35.535274 14954 ts_tablet_manager.cc:1398] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:35.535308 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c92c6475f5ed4529b1efd12b6d495a11 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.535349 11249 consensus_queue.cc:237] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:35.535506 14954 tablet_bootstrap.cc:492] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:35.536413 14954 tablet_bootstrap.cc:654] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.537266 14949 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:35.537370 14949 ts_tablet_manager.cc:1367] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240501 19:56:35.537842 14949 raft_consensus.cc:348] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:35.537957 14949 raft_consensus.cc:374] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.537987 14949 raft_consensus.cc:729] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.538118 14949 consensus_queue.cc:260] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:35.538378 14949 ts_tablet_manager.cc:1398] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:35.538548 14949 tablet_bootstrap.cc:492] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:35.539142 11099 catalog_manager.cc:5484] T c92c6475f5ed4529b1efd12b6d495a11 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.539996 11377 raft_consensus.cc:1261] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.541291 14937 consensus_queue.cc:1035] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.541735 14948 tablet_bootstrap.cc:492] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:35.541872 14948 ts_tablet_manager.cc:1367] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 19:56:35.542145 14935 raft_consensus.cc:482] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.542187 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet c92c6475f5ed4529b1efd12b6d495a11 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:35.542234 14935 raft_consensus.cc:504] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:35.542431 14711 tablet_replica.cc:330] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:35.542426 14948 raft_consensus.cc:348] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.542534 14948 raft_consensus.cc:374] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.542567 14948 raft_consensus.cc:729] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.542560 14711 raft_consensus.cc:2229] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.542672 14711 pending_rounds.cc:62] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: Trying to abort 1 pending ops.
I20240501 19:56:35.542704 14935 leader_election.cc:288] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:35.542717 14948 consensus_queue.cc:260] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.542732 14711 pending_rounds.cc:68] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022974547942789120 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c92c6475f5ed4529b1efd12b6d495a11" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } } }
I20240501 19:56:35.542866 14711 raft_consensus.cc:2894] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 19:56:35.542920 14711 raft_consensus.cc:2866] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:35.542958 14711 raft_consensus.cc:2258] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.543538 14949 tablet_bootstrap.cc:654] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.543931 14948 ts_tablet_manager.cc:1398] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:35.544147 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8078e0cbffa943c98584224d05b0ece0" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:35.544275 11643 raft_consensus.cc:2454] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:35.544641 14711 ts_tablet_manager.cc:1875] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.545271 13735 leader_election.cc:302] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:35.545428 14935 raft_consensus.cc:2781] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.545468 14935 raft_consensus.cc:482] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.545501 14935 raft_consensus.cc:3037] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.546434 14935 raft_consensus.cc:504] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:35.546775 14935 leader_election.cc:288] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:35.546914 14711 ts_tablet_manager.cc:1888] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.546974 14711 log.cc:1196] T c92c6475f5ed4529b1efd12b6d495a11 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/c92c6475f5ed4529b1efd12b6d495a11
I20240501 19:56:35.547173 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8078e0cbffa943c98584224d05b0ece0" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:35.547281 11643 raft_consensus.cc:3037] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.547631 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet c92c6475f5ed4529b1efd12b6d495a11 (table rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]) successfully deleted
I20240501 19:56:35.548344 11643 raft_consensus.cc:2454] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:35.548630 13735 leader_election.cc:302] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:35.548799 14935 raft_consensus.cc:2781] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.548671 14063 raft_consensus.cc:2932] T c92c6475f5ed4529b1efd12b6d495a11 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.548861 14935 raft_consensus.cc:686] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:35.548985 14935 consensus_queue.cc:237] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:35.550235 14943 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.551641 11099 catalog_manager.cc:5484] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.551530 11377 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.552273 14952 tablet_bootstrap.cc:492] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:35.552356 14952 ts_tablet_manager.cc:1367] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.000s
I20240501 19:56:35.552775 14952 raft_consensus.cc:348] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.552888 14952 raft_consensus.cc:374] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.552969 14952 raft_consensus.cc:729] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.553031 13931 raft_consensus.cc:1261] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.553100 14952 consensus_queue.cc:260] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.553377 14952 ts_tablet_manager.cc:1398] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:35.554165 13434 consensus_queue.cc:1035] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.554797 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6070b62336054ef4b0444f27844b9ee1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.555521 13931 raft_consensus.cc:2932] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.557917 14949 tablet_bootstrap.cc:492] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:35.558007 14949 ts_tablet_manager.cc:1367] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240501 19:56:35.557916 11100 catalog_manager.cc:5484] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.558458 14949 raft_consensus.cc:348] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.558550 14949 raft_consensus.cc:374] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.558578 14949 raft_consensus.cc:729] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.558676 14949 consensus_queue.cc:260] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.558934 14949 ts_tablet_manager.cc:1398] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:35.561915 14794 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.562006 14794 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.562343 14794 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:35.562532 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet 6070b62336054ef4b0444f27844b9ee1 with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:35.563270 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc692e320d94825964040105a3175cd" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:35.563403 14063 raft_consensus.cc:2454] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:35.563410 14258 tablet_replica.cc:330] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:35.563594 14258 raft_consensus.cc:2229] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.563721 13734 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:35.563772 14258 raft_consensus.cc:2258] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.563859 14794 raft_consensus.cc:2781] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.563900 14794 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.563930 14794 raft_consensus.cc:3037] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.564496 14258 ts_tablet_manager.cc:1875] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.564882 14794 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.565179 14794 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:35.566946 14258 ts_tablet_manager.cc:1888] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.567011 14258 log.cc:1196] T 6070b62336054ef4b0444f27844b9ee1 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/6070b62336054ef4b0444f27844b9ee1
I20240501 19:56:35.567740 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 6070b62336054ef4b0444f27844b9ee1 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:35.568683 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc692e320d94825964040105a3175cd" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:35.568802 14063 raft_consensus.cc:3037] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.569692 14063 raft_consensus.cc:2454] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:35.570044 13734 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:35.570245 14794 raft_consensus.cc:2781] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.570322 14794 raft_consensus.cc:686] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:35.570466 14794 consensus_queue.cc:237] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.573642 11099 catalog_manager.cc:5484] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.575974 14598 tablet_bootstrap.cc:492] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:35.578212 14598 tablet_bootstrap.cc:492] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:35.578524 14598 ts_tablet_manager.cc:1367] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.063s user 0.024s sys 0.000s
I20240501 19:56:35.579128 14598 raft_consensus.cc:348] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:35.579336 14598 raft_consensus.cc:729] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:35.579528 14598 consensus_queue.cc:260] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:35.579989 14598 ts_tablet_manager.cc:1398] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:35.580415 14976 ts_tablet_manager.cc:902] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:35.580596 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-85ba57eb6aae4f12b4a4641d633104a0 received from {username='slave'} at 127.5.40.7:41129
I20240501 19:56:35.580670 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-85ba57eb6aae4f12b4a4641d633104a0 on tablet 85ba57eb6aae4f12b4a4641d633104a0 with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:35.580894 13670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8078e0cbffa943c98584224d05b0ece0" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:35.581033 13670 raft_consensus.cc:2454] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:35.581262 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8078e0cbffa943c98584224d05b0ece0" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:35.581357 13669 raft_consensus.cc:3037] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.581599 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc692e320d94825964040105a3175cd" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:35.581808 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc692e320d94825964040105a3175cd" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:35.582178 14976 tablet_copy_client.cc:308] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:35.582648 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 4dc0294d0dc5427e80411be5db9c95c1 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:51789)
I20240501 19:56:35.582746 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 4dc0294d0dc5427e80411be5db9c95c1 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:51789: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-4dc0294d0dc5427e80411be5db9c95c1
W20240501 19:56:35.582711 13734 leader_election.cc:341] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Illegal state: must be running to vote when last-logged opid is not known
W20240501 19:56:35.582942 13734 leader_election.cc:341] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:35.583739 11533 tablet_copy_source_session.cc:205] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:35.584057 14954 tablet_bootstrap.cc:492] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:35.584154 14954 ts_tablet_manager.cc:1367] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.000s
I20240501 19:56:35.584421 13669 raft_consensus.cc:2454] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:35.584611 14954 raft_consensus.cc:348] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.584702 14954 raft_consensus.cc:374] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.584728 14954 raft_consensus.cc:729] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.584847 14954 consensus_queue.cc:260] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.584926 14976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc0294d0dc5427e80411be5db9c95c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.585137 14954 ts_tablet_manager.cc:1398] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:35.585412 14954 tablet_bootstrap.cc:492] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:35.586309 14954 tablet_bootstrap.cc:654] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.587069 14976 tablet_copy_client.cc:783] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:35.587209 14976 tablet_copy_client.cc:647] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:35.587301 14954 tablet_bootstrap.cc:492] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:35.587430 14954 ts_tablet_manager.cc:1367] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:35.588011 14954 raft_consensus.cc:348] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.588169 14954 raft_consensus.cc:374] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.588243 14954 raft_consensus.cc:729] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.588557 14374 raft_consensus.cc:482] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.588654 14374 raft_consensus.cc:504] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.589064 14374 leader_election.cc:288] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:35.589550 11100 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:35.589670 11100 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:35.589715 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3883b8a45041328ea220184db9f8ce" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:35.588438 14954 consensus_queue.cc:260] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.590195 13669 raft_consensus.cc:2454] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:35.590342 14976 tablet_copy_client.cc:523] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:35.590526 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3883b8a45041328ea220184db9f8ce" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:35.590646 11508 raft_consensus.cc:2454] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:35.590975 11577 leader_election.cc:302] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:35.591145 14374 raft_consensus.cc:2781] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.591199 14374 raft_consensus.cc:482] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.591235 14374 raft_consensus.cc:3037] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.591727 14976 tablet_bootstrap.cc:492] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:35.592176 14374 raft_consensus.cc:504] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.592489 14374 leader_election.cc:288] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:35.593637 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3883b8a45041328ea220184db9f8ce" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:35.593752 11508 raft_consensus.cc:3037] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.594408 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3883b8a45041328ea220184db9f8ce" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:35.594506 13669 raft_consensus.cc:3037] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.594705 11508 raft_consensus.cc:2454] T 5d3883b8a45041328ea220184db9f8ce P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:35.595099 11100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 9850f9a07d9344a9b39ada629073aee8
I20240501 19:56:35.595422 13669 raft_consensus.cc:2454] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:35.595415 11577 leader_election.cc:302] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:35.595702 14374 raft_consensus.cc:2781] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.595746 13911 tablet_service.cc:1514] Processing DeleteTablet for tablet 9850f9a07d9344a9b39ada629073aee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:35.595780 14374 raft_consensus.cc:686] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:35.595934 14374 consensus_queue.cc:237] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.595988 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet 9850f9a07d9344a9b39ada629073aee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:35.597190 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet 9850f9a07d9344a9b39ada629073aee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:35.597350 14258 tablet_replica.cc:330] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:35.597474 14258 raft_consensus.cc:2229] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.597574 14258 raft_consensus.cc:2258] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.597951 14954 ts_tablet_manager.cc:1398] T 5d3883b8a45041328ea220184db9f8ce P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240501 19:56:35.598009 11100 catalog_manager.cc:5484] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.598354 14258 ts_tablet_manager.cc:1875] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.600450 14258 ts_tablet_manager.cc:1888] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:35.600529 14258 log.cc:1196] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/9850f9a07d9344a9b39ada629073aee8
I20240501 19:56:35.600754 14258 ts_tablet_manager.cc:1909] T 9850f9a07d9344a9b39ada629073aee8 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:35.601331 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 9850f9a07d9344a9b39ada629073aee8 (table rebalancer_dynamic_table_new_name [id=ef1f8bc1aa384faa83a6f7224939cd4e]) successfully deleted
I20240501 19:56:35.601650 14995 tablet_replica.cc:330] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:35.601949 14995 raft_consensus.cc:2229] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:35.602475 14995 raft_consensus.cc:2258] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.603000 14994 tablet_replica.cc:330] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:35.603191 14994 raft_consensus.cc:2229] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.603354 14994 raft_consensus.cc:2258] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.603430 14995 ts_tablet_manager.cc:1875] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.603993 14994 ts_tablet_manager.cc:1875] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.605602 14995 ts_tablet_manager.cc:1888] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:35.605679 14995 log.cc:1196] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/9850f9a07d9344a9b39ada629073aee8
I20240501 19:56:35.605718 14994 ts_tablet_manager.cc:1888] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:35.605839 14994 log.cc:1196] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/9850f9a07d9344a9b39ada629073aee8
I20240501 19:56:35.605983 14995 ts_tablet_manager.cc:1909] T 9850f9a07d9344a9b39ada629073aee8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:35.606142 14994 ts_tablet_manager.cc:1909] T 9850f9a07d9344a9b39ada629073aee8 P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:35.606830 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet 9850f9a07d9344a9b39ada629073aee8 (table rebalancer_dynamic_table_new_name [id=ef1f8bc1aa384faa83a6f7224939cd4e]) successfully deleted
I20240501 19:56:35.606904 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet 9850f9a07d9344a9b39ada629073aee8 (table rebalancer_dynamic_table_new_name [id=ef1f8bc1aa384faa83a6f7224939cd4e]) successfully deleted
I20240501 19:56:35.612356 13801 raft_consensus.cc:1203] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.618885 14937 raft_consensus.cc:1050] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:35.619362 14937 consensus_queue.cc:237] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.620962 11377 raft_consensus.cc:1261] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.621418 13434 consensus_queue.cc:1035] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.621583 14063 raft_consensus.cc:1261] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.621592 11508 raft_consensus.cc:1261] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.622232 14199 consensus_queue.cc:1035] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.622555 14199 consensus_queue.cc:1035] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.624490 13434 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.624890 14280 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.624855 14063 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.624992 14280 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:35.625397 14280 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:35.625978 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc692e320d94825964040105a3175cd" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:35.626310 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc692e320d94825964040105a3175cd" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:35.626446 14063 raft_consensus.cc:2379] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8125d8ad6d0d44caae9c5c90c90df96b in current term 1: Already voted for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in this term.
I20240501 19:56:35.626736 13603 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:35.627637 11377 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.628381 14280 raft_consensus.cc:3037] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.629340 14280 raft_consensus.cc:2726] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:35.629508 11099 catalog_manager.cc:5484] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.630400 11508 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.634696 11249 consensus_queue.cc:237] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.636250 11377 raft_consensus.cc:1261] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.636616 14943 consensus_queue.cc:1035] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.636776 14391 raft_consensus.cc:1050] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:35.637019 14063 raft_consensus.cc:1261] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.637290 14391 consensus_queue.cc:237] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:35.637367 14943 consensus_queue.cc:1035] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.638366 14943 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.638782 13669 raft_consensus.cc:1261] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.639555 13155 consensus_queue.cc:1035] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.640220 11378 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.641465 11100 catalog_manager.cc:5484] T 0419fba851684c9b9cf0e6096de78a03 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.642509 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0419fba851684c9b9cf0e6096de78a03 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.642805 11377 raft_consensus.cc:1261] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.643186 13437 consensus_queue.cc:1035] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.643302 11249 raft_consensus.cc:1261] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.643678 13437 consensus_queue.cc:1035] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.644495 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 0419fba851684c9b9cf0e6096de78a03 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:35.644688 14711 tablet_replica.cc:330] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:35.644840 14711 raft_consensus.cc:2229] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.644986 14711 raft_consensus.cc:2258] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.644985 14063 raft_consensus.cc:2932] T 0419fba851684c9b9cf0e6096de78a03 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.646136 14711 ts_tablet_manager.cc:1875] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.648298 14711 ts_tablet_manager.cc:1888] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.648370 14711 log.cc:1196] T 0419fba851684c9b9cf0e6096de78a03 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/0419fba851684c9b9cf0e6096de78a03
I20240501 19:56:35.648913 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 0419fba851684c9b9cf0e6096de78a03 (table rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]) successfully deleted
I20240501 19:56:35.649749 13437 raft_consensus.cc:2932] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.650202 11249 raft_consensus.cc:2932] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.652541 11105 catalog_manager.cc:5484] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.652748 13669 raft_consensus.cc:2932] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.661676 11508 consensus_queue.cc:237] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:35.662829 13669 raft_consensus.cc:1261] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.662846 11249 raft_consensus.cc:1261] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.663332 15028 consensus_queue.cc:1035] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.663530 14391 consensus_queue.cc:1035] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.664966 14391 raft_consensus.cc:2932] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.666450 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f6a547744514cd88d167a0787d8535d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.666980 13669 raft_consensus.cc:2932] T 1f6a547744514cd88d167a0787d8535d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.667632 11105 catalog_manager.cc:5484] T 1f6a547744514cd88d167a0787d8535d P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.669795 11248 raft_consensus.cc:2932] T 1f6a547744514cd88d167a0787d8535d P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.670222 14976 tablet_bootstrap.cc:492] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:35.670533 14976 tablet_bootstrap.cc:492] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:35.670714 14976 ts_tablet_manager.cc:1367] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.079s user 0.023s sys 0.000s
I20240501 19:56:35.671337 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f6a547744514cd88d167a0787d8535d with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:35.671607 14261 tablet_replica.cc:330] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:35.671758 14261 raft_consensus.cc:2229] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.671893 14261 pending_rounds.cc:62] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: Trying to abort 1 pending ops.
I20240501 19:56:35.671959 14261 pending_rounds.cc:68] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022974548529270784 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "1f6a547744514cd88d167a0787d8535d" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } } }
I20240501 19:56:35.672080 14261 raft_consensus.cc:2894] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 19:56:35.672024 14976 raft_consensus.cc:348] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:35.672128 14261 raft_consensus.cc:2866] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:35.672140 14976 raft_consensus.cc:729] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:35.672159 14261 raft_consensus.cc:2258] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.672238 14976 consensus_queue.cc:260] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:35.672533 14976 ts_tablet_manager.cc:1398] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 19:56:35.673110 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-4dc0294d0dc5427e80411be5db9c95c1 received from {username='slave'} at 127.5.40.6:51789
I20240501 19:56:35.673198 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-4dc0294d0dc5427e80411be5db9c95c1 on tablet 4dc0294d0dc5427e80411be5db9c95c1 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:35.673457 14261 ts_tablet_manager.cc:1875] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.675370 14261 ts_tablet_manager.cc:1888] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.675420 14261 log.cc:1196] T 1f6a547744514cd88d167a0787d8535d P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/1f6a547744514cd88d167a0787d8535d
I20240501 19:56:35.675945 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 1f6a547744514cd88d167a0787d8535d (table rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]) successfully deleted
I20240501 19:56:35.697156 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:35.702749 11228 tablet_service.cc:1467] Processing CreateTablet for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7]), partition=
I20240501 19:56:35.702749 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7]), partition=
I20240501 19:56:35.702757 13911 tablet_service.cc:1467] Processing CreateTablet for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7]), partition=
I20240501 19:56:35.703001 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.703008 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.703002 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.705158 14954 tablet_bootstrap.cc:492] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:35.705603 15053 tablet_bootstrap.cc:492] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:35.705953 14954 tablet_bootstrap.cc:654] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.706501 15053 tablet_bootstrap.cc:654] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.707623 14954 tablet_bootstrap.cc:492] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:35.707708 14954 ts_tablet_manager.cc:1367] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:35.708165 14954 raft_consensus.cc:348] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:35.708271 14954 raft_consensus.cc:374] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.708300 14954 raft_consensus.cc:729] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.708417 14954 consensus_queue.cc:260] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:35.708710 14954 ts_tablet_manager.cc:1398] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:35.710547 15052 tablet_bootstrap.cc:492] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:35.713071 13434 raft_consensus.cc:1050] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:35.713503 13434 consensus_queue.cc:237] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:35.713845 15052 tablet_bootstrap.cc:654] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.715024 11377 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.715468 14943 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.716724 14598 ts_tablet_manager.cc:902] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:35.716929 15052 tablet_bootstrap.cc:492] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:35.717082 15052 ts_tablet_manager.cc:1367] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:35.717162 14598 tablet_copy_client.cc:308] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:35.717530 15052 raft_consensus.cc:348] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:35.717646 15052 raft_consensus.cc:374] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.717715 15052 raft_consensus.cc:729] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.717689 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet b7073666bc3f4c0a8c1e003fb1ef4fef from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:41129)
I20240501 19:56:35.717788 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet b7073666bc3f4c0a8c1e003fb1ef4fef from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:41129: session id = 6921cf2e41f64d66842885512da8de6c-b7073666bc3f4c0a8c1e003fb1ef4fef
I20240501 19:56:35.717868 15052 consensus_queue.cc:260] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:35.718596 15052 ts_tablet_manager.cc:1398] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:35.718772 11533 tablet_copy_source_session.cc:205] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:35.719648 13669 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.719653 11508 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.720364 14598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7073666bc3f4c0a8c1e003fb1ef4fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.721990 14598 tablet_copy_client.cc:783] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:35.722134 14598 tablet_copy_client.cc:647] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:35.722349 14943 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.722568 14937 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.724313 13155 raft_consensus.cc:1050] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:35.724685 13155 consensus_queue.cc:237] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:35.725036 15053 tablet_bootstrap.cc:492] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:35.725131 15053 ts_tablet_manager.cc:1367] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.000s
I20240501 19:56:35.725600 15053 raft_consensus.cc:348] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:35.725710 15053 raft_consensus.cc:374] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.726011 15053 raft_consensus.cc:729] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.726179 15053 consensus_queue.cc:260] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:35.726490 11377 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.726593 15053 ts_tablet_manager.cc:1398] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:35.727655 15028 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.728477 14943 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.729250 11644 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.730140 15028 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.730643 13669 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.730940 11378 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.731402 11099 catalog_manager.cc:5484] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.731488 13669 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.733299 14598 tablet_copy_client.cc:523] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:35.734450 14598 tablet_bootstrap.cc:492] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:35.736164 15028 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.736227 11508 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.740176 14391 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.741202 14374 raft_consensus.cc:1050] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:35.741740 14374 consensus_queue.cc:237] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:35.742085 13669 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.743333 11644 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.743602 11248 consensus_queue.cc:237] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:35.744644 11377 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.745492 11106 catalog_manager.cc:5484] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.746948 11248 raft_consensus.cc:1261] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.747282 13669 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.747926 14505 consensus_queue.cc:1035] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.748782 13801 raft_consensus.cc:1261] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.749101 14199 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.749560 11508 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.749877 14505 consensus_queue.cc:1035] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.749967 14199 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.751286 11511 consensus_queue.cc:237] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:35.752440 11377 raft_consensus.cc:1261] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.752509 11644 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.752580 13669 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.752772 14191 consensus_queue.cc:1035] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.752866 14391 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.753190 13155 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.753778 14199 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.754797 11100 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 19:56:35.754909 11100 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 19:56:35.755575 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c07d9fc05254a6399afcc54b3efb8ac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.756695 13667 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.758853 14191 raft_consensus.cc:1050] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:35.758813 13155 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.759212 14191 consensus_queue.cc:237] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.759979 11103 catalog_manager.cc:5484] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.760208 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e63472cfff449089d88d18b9e495912 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.760739 11511 raft_consensus.cc:1261] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.760840 11249 raft_consensus.cc:1261] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.760918 11508 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.761216 14063 raft_consensus.cc:1261] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.761768 11644 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.762753 11105 catalog_manager.cc:5484] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.761768 14505 consensus_queue.cc:1035] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 19:56:35.763546 14505 consensus_queue.cc:1035] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.763857 14505 consensus_queue.cc:1035] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.765100 14391 raft_consensus.cc:1050] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:35.765441 14391 consensus_queue.cc:237] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:35.767275 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c07d9fc05254a6399afcc54b3efb8ac with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:35.767521 14261 tablet_replica.cc:330] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:35.767661 14261 raft_consensus.cc:2229] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.767818 14261 raft_consensus.cc:2258] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.768425 13931 raft_consensus.cc:1261] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.768527 14261 ts_tablet_manager.cc:1875] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.769038 13437 consensus_queue.cc:1035] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.769804 11248 raft_consensus.cc:1261] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.770655 14261 ts_tablet_manager.cc:1888] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.770714 14261 log.cc:1196] T 1c07d9fc05254a6399afcc54b3efb8ac P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c07d9fc05254a6399afcc54b3efb8ac
I20240501 19:56:35.771399 15028 consensus_queue.cc:1035] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.771895 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 1c07d9fc05254a6399afcc54b3efb8ac (table rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]) successfully deleted
I20240501 19:56:35.772145 14505 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.772449 14598 tablet_bootstrap.cc:492] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:35.773124 14598 tablet_bootstrap.cc:492] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:35.773334 14598 ts_tablet_manager.cc:1367] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.039s user 0.022s sys 0.003s
I20240501 19:56:35.773406 11249 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.773694 13669 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.774076 11377 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.773977 13801 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.775257 14797 raft_consensus.cc:482] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:35.775341 14797 raft_consensus.cc:504] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:35.775714 14797 leader_election.cc:288] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:35.776139 14598 raft_consensus.cc:348] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:35.776247 14598 raft_consensus.cc:729] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:35.776340 14598 consensus_queue.cc:260] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:35.776631 14598 ts_tablet_manager.cc:1398] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240501 19:56:35.776656 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f35f92a9cc04bc58f58ad1a53d0ac03" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:35.776743 13932 raft_consensus.cc:2454] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:35.776790 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e63472cfff449089d88d18b9e495912 with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:35.777040 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-b7073666bc3f4c0a8c1e003fb1ef4fef received from {username='slave'} at 127.5.40.7:41129
I20240501 19:56:35.777105 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-b7073666bc3f4c0a8c1e003fb1ef4fef on tablet b7073666bc3f4c0a8c1e003fb1ef4fef with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:35.777446 13603 leader_election.cc:302] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:35.777784 13209 raft_consensus.cc:1050] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:35.777971 11099 catalog_manager.cc:5484] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.778254 13209 consensus_queue.cc:237] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:35.778612 11100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 0c0c6b4db2cc467da100045058e7b52b
I20240501 19:56:35.778893 11100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 03cf15fd86404aaab8e93953b06a22cc
I20240501 19:56:35.779062 14797 raft_consensus.cc:2781] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:35.779124 14797 raft_consensus.cc:482] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:35.779161 14797 raft_consensus.cc:3037] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.779168 11100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 4 replicas of tablet 3d99f4d3127348dba949ab5fd4666956
I20240501 19:56:35.779975 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0c6b4db2cc467da100045058e7b52b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:35.780103 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet 03cf15fd86404aaab8e93953b06a22cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:35.780144 14797 raft_consensus.cc:504] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:35.780453 11643 raft_consensus.cc:1261] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.780511 14797 leader_election.cc:288] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:35.780536 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0c6b4db2cc467da100045058e7b52b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:35.780663 11490 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d99f4d3127348dba949ab5fd4666956 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:35.780783 14711 tablet_replica.cc:330] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:35.780906 14711 raft_consensus.cc:2229] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.781028 14711 raft_consensus.cc:2258] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.781040 11228 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0c6b4db2cc467da100045058e7b52b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:35.781136 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d99f4d3127348dba949ab5fd4666956 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:35.781071 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 03cf15fd86404aaab8e93953b06a22cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:35.781574 14711 ts_tablet_manager.cc:1875] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.781685 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f35f92a9cc04bc58f58ad1a53d0ac03" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:35.781786 13931 raft_consensus.cc:3037] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.782377 11644 raft_consensus.cc:1261] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.782713 13931 raft_consensus.cc:2454] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:35.783043 15095 tablet_replica.cc:330] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:35.783172 15095 raft_consensus.cc:2229] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.783151 15028 consensus_queue.cc:1035] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.783303 15095 raft_consensus.cc:2258] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.783550 11246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f35f92a9cc04bc58f58ad1a53d0ac03" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:35.783661 11246 raft_consensus.cc:2454] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:35.783816 15095 ts_tablet_manager.cc:1875] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.783905 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f35f92a9cc04bc58f58ad1a53d0ac03" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:35.783998 11245 raft_consensus.cc:3037] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.783998 13603 leader_election.cc:302] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:35.784886 11245 raft_consensus.cc:2454] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:35.785110 11249 raft_consensus.cc:1261] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.785907 15095 ts_tablet_manager.cc:1888] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.786001 15095 log.cc:1196] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/0c0c6b4db2cc467da100045058e7b52b
I20240501 19:56:35.785984 15094 consensus_queue.cc:1035] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.786247 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d99f4d3127348dba949ab5fd4666956 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:35.786283 15095 ts_tablet_manager.cc:1909] T 0c0c6b4db2cc467da100045058e7b52b P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:35.786444 14995 tablet_replica.cc:330] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:35.786571 14995 raft_consensus.cc:2229] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 19:56:35.786612 14934 consensus_queue.cc:1035] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.786672 14995 raft_consensus.cc:2258] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 19:56:35.786670 14797 raft_consensus.cc:2781] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.786749 14797 raft_consensus.cc:686] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:35.786873 14797 consensus_queue.cc:237] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:35.786965 15095 tablet_replica.cc:330] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:35.787070 15095 raft_consensus.cc:2229] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.787179 15095 raft_consensus.cc:2258] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.787297 14995 ts_tablet_manager.cc:1875] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.787675 15095 ts_tablet_manager.cc:1875] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.789049 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet 03cf15fd86404aaab8e93953b06a22cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:35.789062 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d99f4d3127348dba949ab5fd4666956 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:35 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:35.789235 14258 tablet_replica.cc:330] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:35.789319 14995 ts_tablet_manager.cc:1888] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:35.789366 14258 raft_consensus.cc:2229] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.789386 14995 log.cc:1196] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/3d99f4d3127348dba949ab5fd4666956
I20240501 19:56:35.789505 14258 raft_consensus.cc:2258] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.789615 14995 ts_tablet_manager.cc:1909] T 3d99f4d3127348dba949ab5fd4666956 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:35.789544 14391 raft_consensus.cc:2932] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.789786 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 0c0c6b4db2cc467da100045058e7b52b (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.789976 14258 ts_tablet_manager.cc:1875] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.790146 13667 raft_consensus.cc:1261] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.790521 14934 consensus_queue.cc:1035] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.791651 14191 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.792025 14258 ts_tablet_manager.cc:1888] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.792095 14258 log.cc:1196] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/03cf15fd86404aaab8e93953b06a22cc
I20240501 19:56:35.792330 14258 ts_tablet_manager.cc:1909] T 03cf15fd86404aaab8e93953b06a22cc P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:35.792356 13931 raft_consensus.cc:2932] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.792465 11247 raft_consensus.cc:2932] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.793035 14258 tablet_replica.cc:330] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:35.793161 14258 raft_consensus.cc:2229] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:35.793365 14258 raft_consensus.cc:2258] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.793629 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 03cf15fd86404aaab8e93953b06a22cc (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.793972 14063 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.794098 11245 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.794306 11103 catalog_manager.cc:5484] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.796991 14258 ts_tablet_manager.cc:1875] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.798233 11101 catalog_manager.cc:5484] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.798297 11508 consensus_queue.cc:237] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:35.798942 14258 ts_tablet_manager.cc:1888] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:35.799011 14258 log.cc:1196] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d99f4d3127348dba949ab5fd4666956
I20240501 19:56:35.799254 14258 ts_tablet_manager.cc:1909] T 3d99f4d3127348dba949ab5fd4666956 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:35.799541 11245 raft_consensus.cc:1261] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.799573 13931 raft_consensus.cc:1261] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.799949 15028 consensus_queue.cc:1035] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.800082 13437 consensus_queue.cc:1035] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.800333 15095 ts_tablet_manager.cc:1888] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:35.800439 15095 log.cc:1196] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/3d99f4d3127348dba949ab5fd4666956
I20240501 19:56:35.800668 15095 ts_tablet_manager.cc:1909] T 3d99f4d3127348dba949ab5fd4666956 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:35.801147 15095 tablet_replica.cc:330] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:35.801297 15095 raft_consensus.cc:2229] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.801340 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 3d99f4d3127348dba949ab5fd4666956 (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.801450 15095 raft_consensus.cc:2258] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.801918 15095 ts_tablet_manager.cc:1875] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.802055 11102 catalog_manager.cc:5484] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.802826 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet 3d99f4d3127348dba949ab5fd4666956 (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.802961 15028 raft_consensus.cc:2932] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.803792 15095 ts_tablet_manager.cc:1888] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.803849 15095 log.cc:1196] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/03cf15fd86404aaab8e93953b06a22cc
I20240501 19:56:35.804098 15095 ts_tablet_manager.cc:1909] T 03cf15fd86404aaab8e93953b06a22cc P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:35.804330 11643 consensus_queue.cc:237] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:35.804464 13931 raft_consensus.cc:2932] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.804967 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 3d99f4d3127348dba949ab5fd4666956 (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.805528 13801 raft_consensus.cc:1261] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.806288 11377 raft_consensus.cc:1261] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.806983 15081 consensus_queue.cc:1035] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.807161 14374 consensus_queue.cc:1035] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.807600 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 03cf15fd86404aaab8e93953b06a22cc (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.807720 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea873ec8ced340b4ac082417c9b09aae with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.808285 11102 catalog_manager.cc:5484] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.809787 11642 consensus_queue.cc:237] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.810662 11508 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.811509 14374 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.811707 14063 raft_consensus.cc:1261] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.811607 13801 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.813094 15081 consensus_queue.cc:1035] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.813452 11249 raft_consensus.cc:1261] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.813789 14191 consensus_queue.cc:1035] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.813923 11102 catalog_manager.cc:5484] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.814569 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0fb00c1acda4d98a90b12d877412045 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.814927 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet ea873ec8ced340b4ac082417c9b09aae with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:35.815196 14258 tablet_replica.cc:330] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:35.815321 14258 raft_consensus.cc:2229] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.815217 14191 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.815487 11377 raft_consensus.cc:2932] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.816453 14258 pending_rounds.cc:62] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: Trying to abort 1 pending ops.
I20240501 19:56:35.816540 14258 pending_rounds.cc:68] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022974549054308352 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ea873ec8ced340b4ac082417c9b09aae" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } } }
I20240501 19:56:35.816689 14258 raft_consensus.cc:2894] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 19:56:35.816699 14711 ts_tablet_manager.cc:1888] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.816771 14258 raft_consensus.cc:2866] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:35.816776 14711 log.cc:1196] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/0c0c6b4db2cc467da100045058e7b52b
I20240501 19:56:35.816816 14258 raft_consensus.cc:2258] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.816895 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2ecce0ca7a941dbb98b89b797beb8a1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.817055 14711 ts_tablet_manager.cc:1909] T 0c0c6b4db2cc467da100045058e7b52b P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:35.817142 14063 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.818581 14711 tablet_replica.cc:330] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:35.818557 11249 raft_consensus.cc:2932] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.818722 14711 raft_consensus.cc:2229] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.818840 14711 raft_consensus.cc:2258] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.819280 14711 ts_tablet_manager.cc:1875] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.819522 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 0c0c6b4db2cc467da100045058e7b52b (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.819880 14258 ts_tablet_manager.cc:1875] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.820185 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fb00c1acda4d98a90b12d877412045 with delete_type TABLET_DATA_TOMBSTONED (TS 10ee60ead7b94c93ab749ec16a04b0dc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:35.821429 15028 raft_consensus.cc:1050] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:35.821830 15028 consensus_queue.cc:237] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:35.823055 14258 ts_tablet_manager.cc:1888] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.823173 14258 log.cc:1196] T ea873ec8ced340b4ac082417c9b09aae P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/ea873ec8ced340b4ac082417c9b09aae
I20240501 19:56:35.823230 11644 raft_consensus.cc:1261] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.823616 13155 consensus_queue.cc:1035] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.823691 11106 catalog_manager.cc:5484] T b2ecce0ca7a941dbb98b89b797beb8a1 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.823841 15095 tablet_replica.cc:330] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:35.824110 15095 raft_consensus.cc:2229] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.824345 15095 raft_consensus.cc:2258] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.824386 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet ea873ec8ced340b4ac082417c9b09aae (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:35.824612 14261 tablet_replica.cc:330] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:35.824754 14261 raft_consensus.cc:2229] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.824909 14261 raft_consensus.cc:2258] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.825580 13801 raft_consensus.cc:1261] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.825498 11249 raft_consensus.cc:2932] T b2ecce0ca7a941dbb98b89b797beb8a1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.825950 13437 consensus_queue.cc:1035] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.826319 15095 ts_tablet_manager.cc:1875] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.826736 11245 raft_consensus.cc:1261] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.827802 13437 consensus_queue.cc:1035] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.828325 15095 ts_tablet_manager.cc:1888] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.828341 14261 ts_tablet_manager.cc:1875] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.828388 15095 log.cc:1196] T f0fb00c1acda4d98a90b12d877412045 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/f0fb00c1acda4d98a90b12d877412045
I20240501 19:56:35.828891 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet f0fb00c1acda4d98a90b12d877412045 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:35.830410 14261 ts_tablet_manager.cc:1888] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.830271 14391 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.830484 14261 log.cc:1196] T 5e63472cfff449089d88d18b9e495912 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/5e63472cfff449089d88d18b9e495912
I20240501 19:56:35.830639 11249 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.830906 14261 tablet_replica.cc:330] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:35.831077 14261 raft_consensus.cc:2229] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:35.831311 14261 raft_consensus.cc:2258] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.831849 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 5e63472cfff449089d88d18b9e495912 (table rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]) successfully deleted
I20240501 19:56:35.832201 11642 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.833092 11490 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ecce0ca7a941dbb98b89b797beb8a1 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:35.835678 14711 ts_tablet_manager.cc:1888] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:35.835804 14711 log.cc:1196] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d99f4d3127348dba949ab5fd4666956
I20240501 19:56:35.835822 11105 catalog_manager.cc:5484] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.836197 14711 ts_tablet_manager.cc:1909] T 3d99f4d3127348dba949ab5fd4666956 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:35.836807 14711 tablet_replica.cc:330] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:35.837040 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 3d99f4d3127348dba949ab5fd4666956 (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.837416 14261 ts_tablet_manager.cc:1875] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.837591 14711 raft_consensus.cc:2229] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.838923 14934 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.839519 14261 ts_tablet_manager.cc:1888] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.839725 14711 raft_consensus.cc:2258] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.840162 14261 log.cc:1196] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/0c0c6b4db2cc467da100045058e7b52b
I20240501 19:56:35.840462 14711 ts_tablet_manager.cc:1875] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.840494 14261 ts_tablet_manager.cc:1909] T 0c0c6b4db2cc467da100045058e7b52b P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:35.841120 11249 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.842331 14261 tablet_replica.cc:330] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:35.842497 14261 raft_consensus.cc:2229] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:35.842597 14711 ts_tablet_manager.cc:1888] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.842708 14711 log.cc:1196] T b2ecce0ca7a941dbb98b89b797beb8a1 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/b2ecce0ca7a941dbb98b89b797beb8a1
I20240501 19:56:35.842687 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 0c0c6b4db2cc467da100045058e7b52b (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.842782 14261 raft_consensus.cc:2258] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.843195 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet b2ecce0ca7a941dbb98b89b797beb8a1 (table rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]) successfully deleted
I20240501 19:56:35.843302 14261 ts_tablet_manager.cc:1875] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:35.843547 13667 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.845742 14261 ts_tablet_manager.cc:1888] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:35.844624 11106 catalog_manager.cc:5484] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.846915 11643 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.848711 13801 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.851737 11508 consensus_queue.cc:237] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:35.852937 13801 raft_consensus.cc:1261] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.853349 15028 consensus_queue.cc:1035] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.852123 11377 consensus_queue.cc:237] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:35.854444 11643 raft_consensus.cc:1261] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.855100 14261 log.cc:1196] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/03cf15fd86404aaab8e93953b06a22cc
I20240501 19:56:35.855464 14261 ts_tablet_manager.cc:1909] T 03cf15fd86404aaab8e93953b06a22cc P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:35.855644 13667 raft_consensus.cc:1261] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.855754 15028 consensus_queue.cc:1035] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.855940 11249 raft_consensus.cc:1261] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.856047 13437 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.856463 13801 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.856591 14473 consensus_queue.cc:1035] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.857576 11643 raft_consensus.cc:2932] T 04932055883c412f944e839982815ab6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.858901 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04932055883c412f944e839982815ab6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.858847 11105 catalog_manager.cc:5484] T 04932055883c412f944e839982815ab6 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.859846 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 03cf15fd86404aaab8e93953b06a22cc (table rebalance_test_table_8 [id=6700f5e2f06844b7a41c2cabb0ea4f24]) successfully deleted
I20240501 19:56:35.860781 14473 consensus_queue.cc:1035] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.861582 15094 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.862259 11249 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.863234 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 04932055883c412f944e839982815ab6 with delete_type TABLET_DATA_TOMBSTONED (TS 10ee60ead7b94c93ab749ec16a04b0dc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:35.863679 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f1f6382bcc84fcda444dcfae87b00c1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.864297 15095 tablet_replica.cc:330] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:35.864692 11105 catalog_manager.cc:5484] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.865186 15095 raft_consensus.cc:2229] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.865424 15095 raft_consensus.cc:2258] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.866128 15095 ts_tablet_manager.cc:1875] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.867944 13669 raft_consensus.cc:2932] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.869967 15095 ts_tablet_manager.cc:1888] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.870090 15095 log.cc:1196] T 04932055883c412f944e839982815ab6 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/04932055883c412f944e839982815ab6
I20240501 19:56:35.870258 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1f6382bcc84fcda444dcfae87b00c1 with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:35.870504 14258 tablet_replica.cc:330] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:35.870666 14258 raft_consensus.cc:2229] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.870838 14258 raft_consensus.cc:2258] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.871681 14258 ts_tablet_manager.cc:1875] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.873390 14258 ts_tablet_manager.cc:1888] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.873461 14258 log.cc:1196] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/4f1f6382bcc84fcda444dcfae87b00c1
I20240501 19:56:35.873944 15094 raft_consensus.cc:1050] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:35.874048 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 4f1f6382bcc84fcda444dcfae87b00c1 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:35.874331 15094 consensus_queue.cc:237] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.874998 14191 raft_consensus.cc:1050] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:35.875401 14191 consensus_queue.cc:237] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:35.876670 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 04932055883c412f944e839982815ab6 (table rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]) successfully deleted
I20240501 19:56:35.876749 13800 raft_consensus.cc:1261] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.876777 11377 raft_consensus.cc:1261] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.877133 15081 consensus_queue.cc:1035] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.877578 14374 consensus_queue.cc:1035] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.877770 14063 raft_consensus.cc:1261] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.878052 11508 raft_consensus.cc:1261] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.878582 14934 consensus_queue.cc:1035] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.878769 11245 raft_consensus.cc:1261] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.878892 14934 consensus_queue.cc:1035] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.879635 14934 consensus_queue.cc:1035] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.880697 15081 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.881114 11377 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.881580 11249 raft_consensus.cc:1261] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.881954 15081 consensus_queue.cc:1035] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.883986 11247 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.884178 11100 catalog_manager.cc:5484] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.886001 13801 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.886406 14473 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.888156 11643 consensus_queue.cc:237] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:35.889264 11377 raft_consensus.cc:1261] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.889487 13801 raft_consensus.cc:1261] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.889398 11100 catalog_manager.cc:5484] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.889897 14191 consensus_queue.cc:1035] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.890201 15081 consensus_queue.cc:1035] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.890183 11245 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.890292 14063 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.891820 11508 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.892900 15081 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.893020 11377 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.895121 13800 raft_consensus.cc:2932] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:35.895568 11101 catalog_manager.cc:5484] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.898321 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 582d72e630c84088a6e4a7309ae7e952 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.899926 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 582d72e630c84088a6e4a7309ae7e952 with delete_type TABLET_DATA_TOMBSTONED (TS 10ee60ead7b94c93ab749ec16a04b0dc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:35.900131 15095 tablet_replica.cc:330] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:35.900182 11377 consensus_queue.cc:237] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.900271 15095 raft_consensus.cc:2229] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.900386 15095 raft_consensus.cc:2258] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.901718 11245 raft_consensus.cc:1261] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.901723 15094 raft_consensus.cc:1050] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:35.902101 15094 consensus_queue.cc:237] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.902953 15094 consensus_queue.cc:1035] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.903110 15095 ts_tablet_manager.cc:1875] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.903761 13931 raft_consensus.cc:1203] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.903808 11643 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.903834 14062 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.904500 14473 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.905236 15095 ts_tablet_manager.cc:1888] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.905298 15095 log.cc:1196] T 582d72e630c84088a6e4a7309ae7e952 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/582d72e630c84088a6e4a7309ae7e952
I20240501 19:56:35.905663 14063 raft_consensus.cc:1261] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.905958 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 582d72e630c84088a6e4a7309ae7e952 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:35.906040 14473 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.906461 11245 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.908900 13209 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.908821 11102 catalog_manager.cc:5484] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.909201 13209 consensus_queue.cc:1035] T 70fcac1c11404f51838dcb000f76d737 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.909852 11511 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.910518 14062 raft_consensus.cc:2932] T 70fcac1c11404f51838dcb000f76d737 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.910965 14934 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.911477 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 70fcac1c11404f51838dcb000f76d737 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.913220 14934 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.913681 14063 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.914969 11643 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.916225 11099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:35.917848 11106 catalog_manager.cc:5484] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.918605 11490 tablet_service.cc:1514] Processing DeleteTablet for tablet 70fcac1c11404f51838dcb000f76d737 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:35.919061 14711 tablet_replica.cc:330] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:35.919204 14711 raft_consensus.cc:2229] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.919353 14711 raft_consensus.cc:2258] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.920404 14711 ts_tablet_manager.cc:1875] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.922209 11245 raft_consensus.cc:1261] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.922477 14711 ts_tablet_manager.cc:1888] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.922544 14711 log.cc:1196] T 70fcac1c11404f51838dcb000f76d737 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/70fcac1c11404f51838dcb000f76d737
I20240501 19:56:35.922641 13931 raft_consensus.cc:1261] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:35.922782 14797 consensus_queue.cc:1035] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.923055 14280 consensus_queue.cc:1035] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.923548 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 70fcac1c11404f51838dcb000f76d737 (table rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]) successfully deleted
I20240501 19:56:35.924975 11377 consensus_queue.cc:237] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:35.925760 11102 catalog_manager.cc:5576] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965) does not have the latest schema for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (table rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7]). Expected version 1 got 0
I20240501 19:56:35.926061 11099 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (table rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7]). Expected version 1 got 0
I20240501 19:56:35.926498 11643 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.927115 14063 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.927364 15143 tablet.cc:1722] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:35.927618 15094 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.927812 15143 tablet.cc:1620] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:35.927865 15094 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.928629 11099 catalog_manager.cc:5576] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501) does not have the latest schema for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (table rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7]). Expected version 1 got 0
I20240501 19:56:35.930207 15094 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.930763 14896 tablet.cc:1722] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:35.931063 14896 tablet.cc:1620] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:35.932148 14063 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.932550 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cebd15701ee44b11a2f8804740fb4299 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.935031 11643 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:35.936179 11106 catalog_manager.cc:5484] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.936864 11099 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (table rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7]). Expected version 1 got 0
I20240501 19:56:35.937486 11101 catalog_manager.cc:5576] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965) does not have the latest schema for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (table rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7]). Expected version 1 got 0
I20240501 19:56:35.939244 15144 tablet.cc:1722] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:35.939590 15144 tablet.cc:1620] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: MemRowSet was empty: no flush needed.
I20240501 19:56:35.940099 11105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7] alter complete (version 1)
I20240501 19:56:35.942699 11490 tablet_service.cc:1514] Processing DeleteTablet for tablet cebd15701ee44b11a2f8804740fb4299 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:35.943085 14711 tablet_replica.cc:330] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:35.943228 14711 raft_consensus.cc:2229] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.943356 14711 pending_rounds.cc:62] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: Trying to abort 1 pending ops.
I20240501 19:56:35.943392 14711 pending_rounds.cc:68] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022974549614039040 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "cebd15701ee44b11a2f8804740fb4299" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } } }
I20240501 19:56:35.943547 14711 raft_consensus.cc:2894] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 19:56:35.943605 14711 raft_consensus.cc:2866] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:35.943643 14711 raft_consensus.cc:2258] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.944293 14711 ts_tablet_manager.cc:1875] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.946429 14711 ts_tablet_manager.cc:1888] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.946486 14711 log.cc:1196] T cebd15701ee44b11a2f8804740fb4299 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/cebd15701ee44b11a2f8804740fb4299
I20240501 19:56:35.947229 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet cebd15701ee44b11a2f8804740fb4299 (table rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]) successfully deleted
I20240501 19:56:35.954442 11377 consensus_queue.cc:237] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
W20240501 19:56:35.955947 11313 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6326c660e6cb4eb289634a68fc50c4af. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:35.955966 11643 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:35.956183 11511 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:35.956316 14473 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:35.956570 13209 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:35.958014 14191 raft_consensus.cc:1050] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:35.958298 14473 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:35.958385 14191 consensus_queue.cc:237] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:35.958698 11643 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:35.960857 13931 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.961172 11508 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.961390 14505 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.961931 11102 catalog_manager.cc:5484] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:35.965023 11377 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.965246 14505 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.965548 15081 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.966897 14505 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.967540 11377 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.967850 11508 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.969494 13931 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.971513 11101 catalog_manager.cc:5484] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.971873 11511 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:35.976830 11643 consensus_queue.cc:237] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:35.977090 14374 raft_consensus.cc:1050] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:35.977434 14374 consensus_queue.cc:237] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:35.978330 13931 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.978380 11511 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:35.978951 11377 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.978996 14505 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.979362 13667 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.979403 11511 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:35.979768 15081 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.979907 14191 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:35.980623 15081 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.981134 14191 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:35.982295 14374 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.982720 13800 raft_consensus.cc:1203] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:35.983795 13931 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.985862 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 69facf86bba24f209c06afafc1ba8f7b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.986075 14505 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.986816 11102 catalog_manager.cc:5484] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.987264 11508 consensus_queue.cc:237] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:35.987851 11377 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.988925 11643 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:35.988994 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00b7dd445e2846d5a0e1277e160d46bc. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:35.989080 11508 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:35.989365 13437 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:35.990581 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet 69facf86bba24f209c06afafc1ba8f7b with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:35.990794 14261 tablet_replica.cc:330] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:35.990803 11511 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.990945 14261 raft_consensus.cc:2229] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.990864 13667 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.991106 14261 raft_consensus.cc:2258] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.992764 14261 ts_tablet_manager.cc:1875] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.993309 11102 catalog_manager.cc:5484] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:35.994161 11377 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:35.994869 14261 ts_tablet_manager.cc:1888] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:35.994936 14261 log.cc:1196] T 69facf86bba24f209c06afafc1ba8f7b P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/69facf86bba24f209c06afafc1ba8f7b
I20240501 19:56:35.996017 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 69facf86bba24f209c06afafc1ba8f7b (table rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]) successfully deleted
I20240501 19:56:35.997136 14935 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:35.997890 13437 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:35.998224 13437 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:35.998770 11642 consensus_queue.cc:237] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:36.000564 13667 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:36.000643 14794 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.000952 14191 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:36.001751 11508 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:36.001750 11642 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:36.002171 14374 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:36.002336 11377 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:36.003995 11106 catalog_manager.cc:5484] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.006829 15081 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:36.009171 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e307d45fb57f444ca09d94d59b9bcd64 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:36.009328 13669 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:36.009546 11508 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:36.011397 11106 catalog_manager.cc:5484] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:36.014010 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet e307d45fb57f444ca09d94d59b9bcd64 with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:36.014226 14261 tablet_replica.cc:330] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:36.014367 14261 raft_consensus.cc:2229] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.014492 14261 raft_consensus.cc:2258] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.015517 14261 ts_tablet_manager.cc:1875] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:36.017565 14261 ts_tablet_manager.cc:1888] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:36.017627 14261 log.cc:1196] T e307d45fb57f444ca09d94d59b9bcd64 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/e307d45fb57f444ca09d94d59b9bcd64
I20240501 19:56:36.018117 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet e307d45fb57f444ca09d94d59b9bcd64 (table rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]) successfully deleted
I20240501 19:56:36.018385 11377 consensus_queue.cc:237] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:36.019727 11245 raft_consensus.cc:1261] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.020218 11642 raft_consensus.cc:1261] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.020491 15125 consensus_queue.cc:1035] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.022238 15125 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.023718 14934 consensus_queue.cc:1035] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.025027 11101 catalog_manager.cc:5484] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.025405 11642 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.027650 11245 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.029223 14935 consensus_queue.cc:1035] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:36.030000 11316 consensus_peers.cc:485] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01ed8143e5454a4aaef3e142069f35a2. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.037029 11245 consensus_queue.cc:237] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:36.038585 11508 raft_consensus.cc:1261] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.038585 11642 raft_consensus.cc:1261] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.038944 14937 consensus_queue.cc:1035] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:36.039132 11181 consensus_peers.cc:485] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b57520d53734c318d603dd46e30b190. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.039321 14937 consensus_queue.cc:1035] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.041114 14356 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.041581 11642 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.043253 11101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:36.043287 11508 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.044165 11102 catalog_manager.cc:5484] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.047896 14794 consensus_queue.cc:1035] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.052938 15143 tablet.cc:1722] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:36.053298 15143 tablet.cc:1620] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:36.053442 15144 tablet.cc:1722] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:36.053753 15144 tablet.cc:1620] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: MemRowSet was empty: no flush needed.
I20240501 19:56:36.053977 14896 tablet.cc:1722] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:36.054265 14896 tablet.cc:1620] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:36.059512 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=a3b4cae305f24c25bd1038feffeb4bd7] alter complete (version 2)
I20240501 19:56:36.064821 11508 consensus_queue.cc:237] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:36.066289 11377 raft_consensus.cc:1261] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:36.066336 11445 consensus_peers.cc:485] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe34db1ffe5f4509abb547f88c12a562. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.066437 11642 raft_consensus.cc:1261] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.066673 13437 consensus_queue.cc:1035] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.066849 13155 consensus_queue.cc:1035] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.068629 13155 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:36.069217 11377 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:36.069339 11642 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:36.072124 11099 catalog_manager.cc:5484] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:36.080829 15081 consensus_queue.cc:1035] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.084836 15081 consensus_queue.cc:1035] T 5d3883b8a45041328ea220184db9f8ce P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.102957 11102 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:36.109138 13650 tablet_service.cc:1467] Processing CreateTablet for tablet a210b79583db455d86f2c68ee49aa4b7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.109440 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a210b79583db455d86f2c68ee49aa4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.109966 14043 tablet_service.cc:1467] Processing CreateTablet for tablet a210b79583db455d86f2c68ee49aa4b7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.110226 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a210b79583db455d86f2c68ee49aa4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.111071 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 87aabbb9dc2440f28a3b6944dea38fe7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.111331 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aabbb9dc2440f28a3b6944dea38fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.111670 14954 tablet_bootstrap.cc:492] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:36.111932 13780 tablet_service.cc:1467] Processing CreateTablet for tablet a210b79583db455d86f2c68ee49aa4b7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.112196 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a210b79583db455d86f2c68ee49aa4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.112386 13911 tablet_service.cc:1467] Processing CreateTablet for tablet 87aabbb9dc2440f28a3b6944dea38fe7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.112510 14954 tablet_bootstrap.cc:654] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.112629 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aabbb9dc2440f28a3b6944dea38fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.113431 14954 tablet_bootstrap.cc:492] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:36.113502 14954 ts_tablet_manager.cc:1367] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:36.113838 14954 raft_consensus.cc:348] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:36.113924 14954 raft_consensus.cc:374] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.113942 14954 raft_consensus.cc:729] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.114042 14954 consensus_queue.cc:260] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:36.114300 14954 ts_tablet_manager.cc:1398] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.114348 15242 tablet_bootstrap.cc:492] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:36.114552 14042 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0ec896817246b19f753f94ff3f7af5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.114724 13912 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0ec896817246b19f753f94ff3f7af5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.114809 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0ec896817246b19f753f94ff3f7af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.114918 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0ec896817246b19f753f94ff3f7af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.115154 15242 tablet_bootstrap.cc:654] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.116820 15052 tablet_bootstrap.cc:492] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:36.116905 15244 tablet_bootstrap.cc:492] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:36.117630 15052 tablet_bootstrap.cc:654] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.117717 15244 tablet_bootstrap.cc:654] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.118809 13931 raft_consensus.cc:1203] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:36.119329 15244 tablet_bootstrap.cc:492] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:36.119402 15244 ts_tablet_manager.cc:1367] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:36.119546 15053 tablet_bootstrap.cc:492] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:36.119937 15244 raft_consensus.cc:348] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.120087 15244 raft_consensus.cc:374] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.120141 15244 raft_consensus.cc:729] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.120301 15244 consensus_queue.cc:260] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.120399 15053 tablet_bootstrap.cc:654] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.120604 15244 ts_tablet_manager.cc:1398] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.121440 15053 tablet_bootstrap.cc:492] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:36.121544 15053 ts_tablet_manager.cc:1367] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:36.121982 15053 raft_consensus.cc:348] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.122085 15053 raft_consensus.cc:374] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.122121 15053 raft_consensus.cc:729] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.122224 15053 consensus_queue.cc:260] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.122277 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0ec896817246b19f753f94ff3f7af5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.122498 15053 ts_tablet_manager.cc:1398] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:36.122535 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0ec896817246b19f753f94ff3f7af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.125348 15244 tablet_bootstrap.cc:492] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:36.126175 15244 tablet_bootstrap.cc:654] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.126435 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 87aabbb9dc2440f28a3b6944dea38fe7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.126683 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aabbb9dc2440f28a3b6944dea38fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.128360 15052 tablet_bootstrap.cc:492] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:36.128455 15052 ts_tablet_manager.cc:1367] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:36.128683 15242 tablet_bootstrap.cc:492] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:36.128762 15242 ts_tablet_manager.cc:1367] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:56:36.128935 15052 raft_consensus.cc:348] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.129047 15052 raft_consensus.cc:374] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.129078 15052 raft_consensus.cc:729] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.129115 15242 raft_consensus.cc:348] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:36.129206 15242 raft_consensus.cc:374] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.129236 15242 raft_consensus.cc:729] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.129191 15052 consensus_queue.cc:260] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.129369 15242 consensus_queue.cc:260] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:36.129474 15052 ts_tablet_manager.cc:1398] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.129592 15242 ts_tablet_manager.cc:1398] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:36.129632 15052 tablet_bootstrap.cc:492] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:36.129729 15242 tablet_bootstrap.cc:492] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:36.130394 15052 tablet_bootstrap.cc:654] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.130829 15242 tablet_bootstrap.cc:654] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.131734 15052 tablet_bootstrap.cc:492] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:36.131901 15052 ts_tablet_manager.cc:1367] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:36.132018 15242 tablet_bootstrap.cc:492] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:36.132110 15242 ts_tablet_manager.cc:1367] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:36.132534 15242 raft_consensus.cc:348] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.132629 15242 raft_consensus.cc:374] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.132680 15242 raft_consensus.cc:729] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.132643 15052 raft_consensus.cc:348] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.132731 15052 raft_consensus.cc:374] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.132776 15052 raft_consensus.cc:729] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.132818 15242 consensus_queue.cc:260] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.132882 15052 consensus_queue.cc:260] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.133088 15242 ts_tablet_manager.cc:1398] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.133250 15242 tablet_bootstrap.cc:492] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:36.133267 15052 ts_tablet_manager.cc:1398] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:36.133870 15244 tablet_bootstrap.cc:492] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:36.133958 15244 ts_tablet_manager.cc:1367] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:36.134192 15242 tablet_bootstrap.cc:654] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.134377 15244 raft_consensus.cc:348] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:36.134474 15244 raft_consensus.cc:374] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.134505 15244 raft_consensus.cc:729] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.134610 15244 consensus_queue.cc:260] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:36.134864 15244 ts_tablet_manager.cc:1398] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.136772 15242 tablet_bootstrap.cc:492] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:36.136862 15242 ts_tablet_manager.cc:1367] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:36.137243 15242 raft_consensus.cc:348] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.137323 15242 raft_consensus.cc:374] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.137351 15242 raft_consensus.cc:729] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.137472 15242 consensus_queue.cc:260] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.137688 15242 ts_tablet_manager.cc:1398] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.161767 14794 raft_consensus.cc:482] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.161890 14794 raft_consensus.cc:504] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.162377 14794 leader_election.cc:288] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:36.162711 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aabbb9dc2440f28a3b6944dea38fe7" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:36.162844 13931 raft_consensus.cc:2454] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:36.163127 13734 leader_election.cc:302] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:36.163300 14794 raft_consensus.cc:2781] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.163367 14794 raft_consensus.cc:482] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.163400 14794 raft_consensus.cc:3037] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.164228 14794 raft_consensus.cc:504] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.164500 14794 leader_election.cc:288] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:36.164811 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aabbb9dc2440f28a3b6944dea38fe7" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:36.164937 13931 raft_consensus.cc:3037] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.164964 11102 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:36.165865 13931 raft_consensus.cc:2454] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:36.166203 13734 leader_election.cc:302] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:36.166373 14794 raft_consensus.cc:2781] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.166453 14794 raft_consensus.cc:686] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:36.166591 14794 consensus_queue.cc:237] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.168565 11102 catalog_manager.cc:5484] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:36.170521 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aabbb9dc2440f28a3b6944dea38fe7" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:36.170639 11247 raft_consensus.cc:3037] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.170954 15143 tablet.cc:1722] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:36.171651 11247 raft_consensus.cc:2454] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:36.172051 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aabbb9dc2440f28a3b6944dea38fe7" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:36.172165 11245 raft_consensus.cc:2362] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1. Re-sending same reply.
I20240501 19:56:36.174146 14896 tablet.cc:1722] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:36.174391 15144 tablet.cc:1722] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:36.176095 11102 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=a3b4cae305f24c25bd1038feffeb4bd7] alter complete (version 3)
I20240501 19:56:36.228672 11377 consensus_queue.cc:237] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:36.230253 11642 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.230242 11508 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.230743 14473 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.230973 15125 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:36.231171 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01a10926bce14cf1bd12f21287e08c89. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.232730 14473 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.233166 11508 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.233179 11642 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.236052 11106 catalog_manager.cc:5484] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.259580 14935 raft_consensus.cc:482] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.259704 14935 raft_consensus.cc:504] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.260090 14935 leader_election.cc:288] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:36.260459 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0ec896817246b19f753f94ff3f7af5" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:36.260480 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0ec896817246b19f753f94ff3f7af5" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:36.260599 13931 raft_consensus.cc:2454] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:36.260630 14063 raft_consensus.cc:2454] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:36.260880 13734 leader_election.cc:302] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:36.261026 14935 raft_consensus.cc:2781] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.261076 14935 raft_consensus.cc:482] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.261111 14935 raft_consensus.cc:3037] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.261961 14935 raft_consensus.cc:504] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.262315 14935 leader_election.cc:288] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:36.262661 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0ec896817246b19f753f94ff3f7af5" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:36.262737 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0ec896817246b19f753f94ff3f7af5" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:36.262779 13931 raft_consensus.cc:3037] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.262843 14063 raft_consensus.cc:3037] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.263762 14063 raft_consensus.cc:2454] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:36.263762 13931 raft_consensus.cc:2454] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:36.264070 13734 leader_election.cc:302] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:36.265247 14935 raft_consensus.cc:2781] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.265332 14935 raft_consensus.cc:686] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:36.265468 14935 consensus_queue.cc:237] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:36.267314 11102 catalog_manager.cc:5484] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:36.281286 11102 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 19:56:36.281442 11102 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 19:56:36.281246 11097 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:36.281693 11097 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:36.284442 11102 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 6070b62336054ef4b0444f27844b9ee1
I20240501 19:56:36.284752 11102 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet c7cd411ab06242d7809d56cd138fd7a5
I20240501 19:56:36.284981 11102 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet f0fb00c1acda4d98a90b12d877412045
I20240501 19:56:36.285307 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet c7cd411ab06242d7809d56cd138fd7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:36.285348 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet 6070b62336054ef4b0444f27844b9ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:36.285678 14261 tablet_replica.cc:330] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:36.285764 14258 tablet_replica.cc:330] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:36.285741 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 6070b62336054ef4b0444f27844b9ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:36.285759 11490 tablet_service.cc:1514] Processing DeleteTablet for tablet c7cd411ab06242d7809d56cd138fd7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:36.285828 14261 raft_consensus.cc:2229] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.285902 14258 raft_consensus.cc:2229] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:36.285909 11228 tablet_service.cc:1514] Processing DeleteTablet for tablet c7cd411ab06242d7809d56cd138fd7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:36.285948 15095 tablet_replica.cc:330] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:36.286013 14261 raft_consensus.cc:2258] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.286087 15095 raft_consensus.cc:2229] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:36.286096 14258 raft_consensus.cc:2258] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.286276 15095 raft_consensus.cc:2258] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.286664 14258 ts_tablet_manager.cc:1875] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.287113 14261 ts_tablet_manager.cc:1875] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.287235 15095 ts_tablet_manager.cc:1875] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.288815 14258 ts_tablet_manager.cc:1888] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.288864 14258 log.cc:1196] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/c7cd411ab06242d7809d56cd138fd7a5
I20240501 19:56:36.289032 14258 ts_tablet_manager.cc:1909] T c7cd411ab06242d7809d56cd138fd7a5 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:36.289197 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet 6070b62336054ef4b0444f27844b9ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:36.289232 15095 ts_tablet_manager.cc:1888] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:36.289255 14261 ts_tablet_manager.cc:1888] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:36.289296 15095 log.cc:1196] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/6070b62336054ef4b0444f27844b9ee1
I20240501 19:56:36.289310 14261 log.cc:1196] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/6070b62336054ef4b0444f27844b9ee1
I20240501 19:56:36.289465 14711 tablet_replica.cc:330] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:36.289534 14261 ts_tablet_manager.cc:1909] T 6070b62336054ef4b0444f27844b9ee1 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:36.289534 15095 ts_tablet_manager.cc:1909] T 6070b62336054ef4b0444f27844b9ee1 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:36.289613 14711 raft_consensus.cc:2229] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.289732 14711 raft_consensus.cc:2258] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.290128 14711 ts_tablet_manager.cc:1875] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.290138 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fb00c1acda4d98a90b12d877412045 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:36.290339 14995 tablet_replica.cc:330] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:36.290437 14995 raft_consensus.cc:2229] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.290536 14995 raft_consensus.cc:2258] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.291124 14995 ts_tablet_manager.cc:1875] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.291373 11097 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 1f35f92a9cc04bc58f58ad1a53d0ac03
I20240501 19:56:36.292071 15095 tablet_replica.cc:330] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:36.292161 14711 ts_tablet_manager.cc:1888] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.292201 15095 raft_consensus.cc:2229] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.292219 14711 log.cc:1196] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/c7cd411ab06242d7809d56cd138fd7a5
I20240501 19:56:36.292316 15095 raft_consensus.cc:2258] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.292450 14711 ts_tablet_manager.cc:1909] T c7cd411ab06242d7809d56cd138fd7a5 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:36.292768 15095 ts_tablet_manager.cc:1875] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.292805 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 6070b62336054ef4b0444f27844b9ee1 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:36.293056 14995 ts_tablet_manager.cc:1888] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:36.293118 14995 log.cc:1196] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/f0fb00c1acda4d98a90b12d877412045
I20240501 19:56:36.293354 14995 ts_tablet_manager.cc:1909] T f0fb00c1acda4d98a90b12d877412045 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:36.294178 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fb00c1acda4d98a90b12d877412045 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:36.294377 13650 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:36.294484 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:36.294713 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet c7cd411ab06242d7809d56cd138fd7a5 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:36.294878 15366 tablet_replica.cc:330] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:36.294999 15366 raft_consensus.cc:2229] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:36.295094 15095 ts_tablet_manager.cc:1888] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.295171 15095 log.cc:1196] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/c7cd411ab06242d7809d56cd138fd7a5
I20240501 19:56:36.295248 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fb00c1acda4d98a90b12d877412045 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:36.295367 11228 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:36.295387 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 6070b62336054ef4b0444f27844b9ee1 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:36.295431 14261 tablet_replica.cc:330] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:36.295537 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet f0fb00c1acda4d98a90b12d877412045 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:36.295570 14261 raft_consensus.cc:2229] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.295732 14261 raft_consensus.cc:2258] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.295789 15095 ts_tablet_manager.cc:1909] T c7cd411ab06242d7809d56cd138fd7a5 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:36.296350 14261 ts_tablet_manager.cc:1875] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.296372 15095 tablet_replica.cc:330] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:36.296485 15095 raft_consensus.cc:2229] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.296587 15095 raft_consensus.cc:2258] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.297205 15095 ts_tablet_manager.cc:1875] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.298378 14261 ts_tablet_manager.cc:1888] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:36.298380 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet c7cd411ab06242d7809d56cd138fd7a5 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:36.298454 14261 log.cc:1196] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0fb00c1acda4d98a90b12d877412045
I20240501 19:56:36.298801 14261 ts_tablet_manager.cc:1909] T f0fb00c1acda4d98a90b12d877412045 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:36.299105 15095 ts_tablet_manager.cc:1888] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:36.299222 15095 log.cc:1196] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/1f35f92a9cc04bc58f58ad1a53d0ac03
I20240501 19:56:36.299432 15366 raft_consensus.cc:2258] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.299548 15095 ts_tablet_manager.cc:1909] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:36.299522 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet f0fb00c1acda4d98a90b12d877412045 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:36.300107 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (table rebalancer_dynamic_table_new_name [id=a3b4cae305f24c25bd1038feffeb4bd7]) successfully deleted
I20240501 19:56:36.300312 15366 ts_tablet_manager.cc:1875] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.300379 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet c7cd411ab06242d7809d56cd138fd7a5 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:36.300750 14258 tablet_replica.cc:330] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:36.300930 14258 raft_consensus.cc:2229] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:36.301172 14258 raft_consensus.cc:2258] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.301875 14258 ts_tablet_manager.cc:1875] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.302696 15366 ts_tablet_manager.cc:1888] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:36.302816 15366 log.cc:1196] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/1f35f92a9cc04bc58f58ad1a53d0ac03
I20240501 19:56:36.303133 15366 ts_tablet_manager.cc:1909] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting consensus metadata
I20240501 19:56:36.303639 15365 tablet_replica.cc:330] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:36.303828 15365 raft_consensus.cc:2229] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.303833 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (table rebalancer_dynamic_table_new_name [id=a3b4cae305f24c25bd1038feffeb4bd7]) successfully deleted
I20240501 19:56:36.303977 14258 ts_tablet_manager.cc:1888] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:36.304054 14258 log.cc:1196] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/f0fb00c1acda4d98a90b12d877412045
I20240501 19:56:36.304047 15365 raft_consensus.cc:2258] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.304334 14258 ts_tablet_manager.cc:1909] T f0fb00c1acda4d98a90b12d877412045 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:36.304755 15365 ts_tablet_manager.cc:1875] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.304983 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet f0fb00c1acda4d98a90b12d877412045 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:36.308085 15365 ts_tablet_manager.cc:1888] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:36.308173 15365 log.cc:1196] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/6070b62336054ef4b0444f27844b9ee1
I20240501 19:56:36.308432 15365 ts_tablet_manager.cc:1909] T 6070b62336054ef4b0444f27844b9ee1 P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:36.309036 15365 tablet_replica.cc:330] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:36.309089 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet 6070b62336054ef4b0444f27844b9ee1 (table rebalance_test_table_10 [id=971529fa9d024a01b952770cd989c2d9]) successfully deleted
I20240501 19:56:36.309193 15365 raft_consensus.cc:2229] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.309319 15365 raft_consensus.cc:2258] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.309868 15365 ts_tablet_manager.cc:1875] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.311774 15365 ts_tablet_manager.cc:1888] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:36.311902 15365 log.cc:1196] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/1f35f92a9cc04bc58f58ad1a53d0ac03
I20240501 19:56:36.312201 15365 ts_tablet_manager.cc:1909] T 1f35f92a9cc04bc58f58ad1a53d0ac03 P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:36.312765 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet 1f35f92a9cc04bc58f58ad1a53d0ac03 (table rebalancer_dynamic_table_new_name [id=a3b4cae305f24c25bd1038feffeb4bd7]) successfully deleted
I20240501 19:56:36.346122 15386 ts_tablet_manager.cc:902] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:36.346560 15386 tablet_copy_client.cc:308] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:36.347002 11400 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet 6326c660e6cb4eb289634a68fc50c4af from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:36785)
I20240501 19:56:36.347113 11400 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet 6326c660e6cb4eb289634a68fc50c4af from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:36785: session id = 8125d8ad6d0d44caae9c5c90c90df96b-6326c660e6cb4eb289634a68fc50c4af
I20240501 19:56:36.347946 11400 tablet_copy_source_session.cc:205] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:36.348762 15386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6326c660e6cb4eb289634a68fc50c4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.350574 15386 tablet_copy_client.cc:783] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:36.350735 15386 tablet_copy_client.cc:647] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:36.351881 15386 tablet_copy_client.cc:523] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:36.352913 15386 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:36.373442 15386 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:36.373739 15386 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:36.373917 15386 ts_tablet_manager.cc:1367] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240501 19:56:36.374353 15386 raft_consensus.cc:348] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:36.374471 15386 raft_consensus.cc:729] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:36.374565 15386 consensus_queue.cc:260] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:36.374836 15386 ts_tablet_manager.cc:1398] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:36.375730 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-6326c660e6cb4eb289634a68fc50c4af received from {username='slave'} at 127.5.40.5:36785
I20240501 19:56:36.375814 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-6326c660e6cb4eb289634a68fc50c4af on tablet 6326c660e6cb4eb289634a68fc50c4af with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:36.388188 13437 raft_consensus.cc:1050] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:36.388576 13437 consensus_queue.cc:237] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:36.390157 11377 raft_consensus.cc:1261] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:36.390233 13800 raft_consensus.cc:1261] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:36.390538 11245 raft_consensus.cc:1261] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:36.390767 15393 ts_tablet_manager.cc:902] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:36.390859 14391 consensus_queue.cc:1035] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.391202 14391 consensus_queue.cc:1035] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.391239 15393 tablet_copy_client.cc:308] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:36.391752 13155 consensus_queue.cc:1035] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.392315 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 00b7dd445e2846d5a0e1277e160d46bc from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:41129)
I20240501 19:56:36.392410 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 00b7dd445e2846d5a0e1277e160d46bc from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:41129: session id = 6921cf2e41f64d66842885512da8de6c-00b7dd445e2846d5a0e1277e160d46bc
I20240501 19:56:36.393347 11533 tablet_copy_source_session.cc:205] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:36.393779 15028 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:36.394061 13800 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:36.394055 11245 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:36.394313 15393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b7dd445e2846d5a0e1277e160d46bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.395629 11377 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:36.397123 11100 catalog_manager.cc:5484] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:36.399893 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:36.400707 15393 tablet_copy_client.cc:783] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:36.400875 15393 tablet_copy_client.cc:647] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:36.401541 11508 consensus_queue.cc:237] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:36.403474 13800 raft_consensus.cc:1261] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:36.403825 11377 raft_consensus.cc:1261] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:36.403892 14391 consensus_queue.cc:1035] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:36.404224 13155 consensus_queue.cc:1035] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:36.406011 15028 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:36.406388 11377 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:36.406509 13800 raft_consensus.cc:2932] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:36.408309 15393 tablet_copy_client.cc:523] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:36.408950 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dc0294d0dc5427e80411be5db9c95c1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:36.409560 13912 tablet_service.cc:1467] Processing CreateTablet for tablet 87475cc3ef6b4831889cce01290a70ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=63f9f6bda41a42779756e2f9fa74e4d5]), partition=
I20240501 19:56:36.409703 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 87475cc3ef6b4831889cce01290a70ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=63f9f6bda41a42779756e2f9fa74e4d5]), partition=
I20240501 19:56:36.409780 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87475cc3ef6b4831889cce01290a70ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.409973 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87475cc3ef6b4831889cce01290a70ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.410027 15393 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:36.411458 15052 tablet_bootstrap.cc:492] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:36.411666 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 87475cc3ef6b4831889cce01290a70ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=63f9f6bda41a42779756e2f9fa74e4d5]), partition=
I20240501 19:56:36.411896 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87475cc3ef6b4831889cce01290a70ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.412230 15244 tablet_bootstrap.cc:492] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:36.412314 15052 tablet_bootstrap.cc:654] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.413113 15244 tablet_bootstrap.cc:654] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.415012 11106 catalog_manager.cc:5484] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:36.415628 14954 tablet_bootstrap.cc:492] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:36.416317 14954 tablet_bootstrap.cc:654] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:36.417994 11228 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc0294d0dc5427e80411be5db9c95c1 with delete_type TABLET_DATA_TOMBSTONED (TS 10ee60ead7b94c93ab749ec16a04b0dc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:36.418195 15095 tablet_replica.cc:330] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:36.418334 15095 raft_consensus.cc:2229] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.418493 15095 raft_consensus.cc:2258] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.418668 14954 tablet_bootstrap.cc:492] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:36.418751 14954 ts_tablet_manager.cc:1367] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:36.419180 14954 raft_consensus.cc:348] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:36.419291 14954 raft_consensus.cc:374] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.419322 14954 raft_consensus.cc:729] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.419430 14954 consensus_queue.cc:260] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:36.419716 14954 ts_tablet_manager.cc:1398] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:36.422605 15052 tablet_bootstrap.cc:492] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:36.422698 15052 ts_tablet_manager.cc:1367] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240501 19:56:36.422938 15095 ts_tablet_manager.cc:1875] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:36.423139 15052 raft_consensus.cc:348] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:36.423252 15052 raft_consensus.cc:374] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.423293 15052 raft_consensus.cc:729] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.423420 15052 consensus_queue.cc:260] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:36.423729 15052 ts_tablet_manager.cc:1398] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:36.425014 15244 tablet_bootstrap.cc:492] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:36.425105 15244 ts_tablet_manager.cc:1367] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 19:56:36.425163 15095 ts_tablet_manager.cc:1888] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:36.425231 15095 log.cc:1196] T 4dc0294d0dc5427e80411be5db9c95c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/4dc0294d0dc5427e80411be5db9c95c1
I20240501 19:56:36.425540 15244 raft_consensus.cc:348] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:36.425670 15244 raft_consensus.cc:374] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:36.425702 15244 raft_consensus.cc:729] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:36.425825 15244 consensus_queue.cc:260] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:36.425905 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 4dc0294d0dc5427e80411be5db9c95c1 (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:36.426081 15244 ts_tablet_manager.cc:1398] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.436066 15393 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:36.436367 15393 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:36.436560 15393 ts_tablet_manager.cc:1367] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.005s
I20240501 19:56:36.437013 15393 raft_consensus.cc:348] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:36.437119 15393 raft_consensus.cc:729] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:36.437187 15393 consensus_queue.cc:260] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:36.437428 15393 ts_tablet_manager.cc:1398] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:36.437896 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-00b7dd445e2846d5a0e1277e160d46bc received from {username='slave'} at 127.5.40.7:41129
I20240501 19:56:36.437983 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-00b7dd445e2846d5a0e1277e160d46bc on tablet 00b7dd445e2846d5a0e1277e160d46bc with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:36.438252 15203 raft_consensus.cc:482] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.438367 15203 raft_consensus.cc:504] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:36.439489 15203 leader_election.cc:288] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:36.444043 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a210b79583db455d86f2c68ee49aa4b7" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:36.444178 13800 raft_consensus.cc:2454] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:36.444506 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a210b79583db455d86f2c68ee49aa4b7" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:36.444540 13999 leader_election.cc:302] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:36.444636 13669 raft_consensus.cc:2454] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:36.444727 15203 raft_consensus.cc:2781] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.444818 15203 raft_consensus.cc:482] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.444869 15203 raft_consensus.cc:3037] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.445657 15203 raft_consensus.cc:504] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:36.445947 15203 leader_election.cc:288] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:36.446226 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a210b79583db455d86f2c68ee49aa4b7" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:36.446305 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a210b79583db455d86f2c68ee49aa4b7" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:36.446343 13669 raft_consensus.cc:3037] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.446406 13800 raft_consensus.cc:3037] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.447285 13800 raft_consensus.cc:2454] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:36.447285 13669 raft_consensus.cc:2454] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:36.447598 13996 leader_election.cc:302] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:36.447750 15203 raft_consensus.cc:2781] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.447853 15203 raft_consensus.cc:686] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:36.448143 15203 consensus_queue.cc:237] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:36.450572 11106 catalog_manager.cc:5484] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:36.468191 15409 ts_tablet_manager.cc:902] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:36.468791 15409 tablet_copy_client.cc:308] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:36.473841 11400 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet 01ed8143e5454a4aaef3e142069f35a2 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:59475)
I20240501 19:56:36.473953 11400 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet 01ed8143e5454a4aaef3e142069f35a2 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:59475: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-01ed8143e5454a4aaef3e142069f35a2
I20240501 19:56:36.474887 11400 tablet_copy_source_session.cc:205] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:36.476071 15409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ed8143e5454a4aaef3e142069f35a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.478385 15409 tablet_copy_client.cc:783] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:36.478535 15409 tablet_copy_client.cc:647] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:36.479825 15409 tablet_copy_client.cc:523] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:36.482867 11490 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 19:56:36.483036 15386 ts_tablet_manager.cc:902] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:36.483559 15386 tablet_copy_client.cc:308] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:36.485478 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet fe34db1ffe5f4509abb547f88c12a562 from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:53821)
I20240501 19:56:36.485569 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet fe34db1ffe5f4509abb547f88c12a562 from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:53821: session id = 8125d8ad6d0d44caae9c5c90c90df96b-fe34db1ffe5f4509abb547f88c12a562
I20240501 19:56:36.486550 11533 tablet_copy_source_session.cc:205] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:36.488893 15386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe34db1ffe5f4509abb547f88c12a562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.489503 13437 raft_consensus.cc:1050] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:36.489917 13437 consensus_queue.cc:237] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:36.491175 15386 tablet_copy_client.cc:783] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:36.491329 15386 tablet_copy_client.cc:647] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:36.491555 11642 raft_consensus.cc:1261] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:36.491914 15028 consensus_queue.cc:1035] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.493286 15409 tablet_bootstrap.cc:492] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:36.493772 13931 raft_consensus.cc:1261] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:36.493867 11247 raft_consensus.cc:1261] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:36.494155 11624 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:56:36.494259 15028 consensus_queue.cc:1035] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.494638 13155 consensus_queue.cc:1035] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.501125 15427 ts_tablet_manager.cc:902] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:36.501175 13912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:36.503585 11358 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:36.505060 15427 tablet_copy_client.cc:308] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:42965
I20240501 19:56:36.505543 11269 tablet_copy_service.cc:140] P 10ee60ead7b94c93ab749ec16a04b0dc: Received BeginTabletCopySession request for tablet 1b57520d53734c318d603dd46e30b190 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:50213)
I20240501 19:56:36.505666 11269 tablet_copy_service.cc:161] P 10ee60ead7b94c93ab749ec16a04b0dc: Beginning new tablet copy session on tablet 1b57520d53734c318d603dd46e30b190 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:50213: session id = 533908e517b24f77b2af8b5863e4ae5a-1b57520d53734c318d603dd46e30b190
I20240501 19:56:36.506536 11269 tablet_copy_source_session.cc:205] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:36.507388 13650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:36.507504 15427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b57520d53734c318d603dd46e30b190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.508988 14391 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.509517 11247 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.512447 11106 catalog_manager.cc:5484] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.512787 11642 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.514844 15427 tablet_copy_client.cc:783] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:36.515040 15427 tablet_copy_client.cc:647] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:36.518786 14043 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:36.518939 11228 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:36.520351 15427 tablet_copy_client.cc:523] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:36.523530 11511 consensus_queue.cc:237] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:36.526300 11245 raft_consensus.cc:1261] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:36.526749 13932 raft_consensus.cc:1261] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:36.526726 14391 consensus_queue.cc:1035] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:36.527189 14391 consensus_queue.cc:1035] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.527346 13931 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.528510 15427 tablet_bootstrap.cc:492] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:36.532065 13437 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.533845 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 85ba57eb6aae4f12b4a4641d633104a0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:36.534734 13931 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.535928 11100 catalog_manager.cc:5484] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:36.536070 13780 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:36.551833 15386 tablet_copy_client.cc:523] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:36.552563 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ba57eb6aae4f12b4a4641d633104a0 with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:36.552884 14258 tablet_replica.cc:330] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:36.553032 14258 raft_consensus.cc:2229] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.553194 14258 raft_consensus.cc:2258] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.553635 15386 tablet_bootstrap.cc:492] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:36.553920 14258 ts_tablet_manager.cc:1875] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:36.556360 14258 ts_tablet_manager.cc:1888] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:36.556625 14258 log.cc:1196] T 85ba57eb6aae4f12b4a4641d633104a0 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/85ba57eb6aae4f12b4a4641d633104a0
I20240501 19:56:36.557247 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 85ba57eb6aae4f12b4a4641d633104a0 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:36.559336 15427 tablet_bootstrap.cc:492] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:36.559756 15427 tablet_bootstrap.cc:492] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:36.559978 15427 ts_tablet_manager.cc:1367] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.000s
I20240501 19:56:36.560482 15427 raft_consensus.cc:348] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:36.560686 15427 raft_consensus.cc:729] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:36.560897 15427 consensus_queue.cc:260] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:36.561337 15427 ts_tablet_manager.cc:1398] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.562433 11269 tablet_copy_service.cc:342] P 10ee60ead7b94c93ab749ec16a04b0dc: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-1b57520d53734c318d603dd46e30b190 received from {username='slave'} at 127.5.40.8:50213
I20240501 19:56:36.562530 11269 tablet_copy_service.cc:433] P 10ee60ead7b94c93ab749ec16a04b0dc: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-1b57520d53734c318d603dd46e30b190 on tablet 1b57520d53734c318d603dd46e30b190 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:36.583302 15409 tablet_bootstrap.cc:492] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:36.583763 15409 tablet_bootstrap.cc:492] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:36.583954 15409 ts_tablet_manager.cc:1367] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.091s user 0.024s sys 0.000s
I20240501 19:56:36.584335 15409 raft_consensus.cc:348] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:36.584532 15409 raft_consensus.cc:729] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:36.584648 15409 consensus_queue.cc:260] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:36.584946 15409 ts_tablet_manager.cc:1398] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.585732 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-01ed8143e5454a4aaef3e142069f35a2 received from {username='slave'} at 127.5.40.6:59475
I20240501 19:56:36.585829 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-01ed8143e5454a4aaef3e142069f35a2 on tablet 01ed8143e5454a4aaef3e142069f35a2 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:36.586373 11245 raft_consensus.cc:2932] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.596055 15386 tablet_bootstrap.cc:492] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:36.596477 15386 tablet_bootstrap.cc:492] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:36.596696 15386 ts_tablet_manager.cc:1367] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.043s user 0.025s sys 0.001s
I20240501 19:56:36.597168 15386 raft_consensus.cc:348] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:36.597607 15386 raft_consensus.cc:729] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:36.597764 15386 consensus_queue.cc:260] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:36.598078 15386 ts_tablet_manager.cc:1398] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.598613 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-fe34db1ffe5f4509abb547f88c12a562 received from {username='slave'} at 127.5.40.5:53821
I20240501 19:56:36.598701 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-fe34db1ffe5f4509abb547f88c12a562 on tablet fe34db1ffe5f4509abb547f88c12a562 with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:36.610888 13155 raft_consensus.cc:1050] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:36.611295 13155 consensus_queue.cc:237] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:36.612907 11245 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:36.613351 11642 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:36.613570 14391 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.613727 13931 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:36.614490 13437 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.614809 13437 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:36.619522 13437 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.620390 11245 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.621752 11642 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.624078 13931 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.624255 11099 catalog_manager.cc:5484] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:36.628916 11511 consensus_queue.cc:237] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:36.630220 13931 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:36.630766 13155 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:36.633908 11245 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:36.634318 13155 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:36.635246 15028 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.636026 13931 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.637549 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b7073666bc3f4c0a8c1e003fb1ef4fef with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:36.639032 11100 catalog_manager.cc:5484] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.639948 11245 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:36.642102 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet b7073666bc3f4c0a8c1e003fb1ef4fef with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:36.642381 14258 tablet_replica.cc:330] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:36.642539 14258 raft_consensus.cc:2229] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.642720 14258 raft_consensus.cc:2258] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.643986 14258 ts_tablet_manager.cc:1875] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:36.646314 14258 ts_tablet_manager.cc:1888] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:36.646407 14258 log.cc:1196] T b7073666bc3f4c0a8c1e003fb1ef4fef P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/b7073666bc3f4c0a8c1e003fb1ef4fef
I20240501 19:56:36.646955 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet b7073666bc3f4c0a8c1e003fb1ef4fef (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:36.648736 11377 consensus_queue.cc:237] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:36.650324 11511 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:36.650689 11313 consensus_peers.cc:485] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 169db26dfa3a4ec991c425c870e669eb. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.650893 15094 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.651301 11642 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.651767 15094 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.653613 15125 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:36.656035 11642 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:36.656421 11100 catalog_manager.cc:5484] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.659670 11511 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:36.671710 11245 consensus_queue.cc:237] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:36.673722 11511 raft_consensus.cc:1261] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.673918 11642 raft_consensus.cc:1261] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.674331 15462 consensus_queue.cc:1035] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.674650 15461 consensus_queue.cc:1035] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.677372 15462 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.677789 11511 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.680675 11100 catalog_manager.cc:5484] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.682695 11642 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
W20240501 19:56:36.685204 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e96e8e7c66584b1198bea464e5a1a4d8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.693073 11245 consensus_queue.cc:237] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:36.694530 11377 raft_consensus.cc:1261] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.694563 11642 raft_consensus.cc:1261] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.694967 13434 consensus_queue.cc:1035] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:36.695163 11181 consensus_peers.cc:485] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c31cf707a7e84f139fce27df22b376fd. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.695188 15462 consensus_queue.cc:1035] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.696985 15461 raft_consensus.cc:2932] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.697487 11377 raft_consensus.cc:2932] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.697553 11642 raft_consensus.cc:2932] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.700161 11100 catalog_manager.cc:5484] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.709918 14618 raft_consensus.cc:482] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:36.709977 11245 consensus_queue.cc:237] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:36.710054 14618 raft_consensus.cc:504] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:36.710469 14618 leader_election.cc:288] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:36.710788 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87475cc3ef6b4831889cce01290a70ce" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:36.710876 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87475cc3ef6b4831889cce01290a70ce" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:36.710928 14063 raft_consensus.cc:2454] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:36.711024 13931 raft_consensus.cc:2454] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:36.711318 13603 leader_election.cc:302] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:36.711606 14618 raft_consensus.cc:2781] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:36.711663 14618 raft_consensus.cc:482] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:36.711717 14618 raft_consensus.cc:3037] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.711743 11511 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.712553 14618 raft_consensus.cc:504] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:36.712879 14618 leader_election.cc:288] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:36.713214 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87475cc3ef6b4831889cce01290a70ce" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:36.713230 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87475cc3ef6b4831889cce01290a70ce" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:36.713332 14063 raft_consensus.cc:3037] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:36.713351 13932 raft_consensus.cc:3037] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:56:36.713609 11181 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3176bda2ad4b9986cda48e4a03ce49. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.713712 15461 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.714349 14063 raft_consensus.cc:2454] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:36.714381 13932 raft_consensus.cc:2454] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:36.714526 11377 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.714660 13603 leader_election.cc:302] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:36.714861 14618 raft_consensus.cc:2781] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:36.714883 15461 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.715032 14618 raft_consensus.cc:686] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:36.715190 14618 consensus_queue.cc:237] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:36.716897 13434 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:36.717321 11511 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:36.719205 11105 catalog_manager.cc:5484] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:36.719727 11100 catalog_manager.cc:5484] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.724129 11377 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:36.731072 15427 ts_tablet_manager.cc:902] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:36.731580 15427 tablet_copy_client.cc:308] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:36.732035 11400 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet 01a10926bce14cf1bd12f21287e08c89 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:39229)
I20240501 19:56:36.732132 11400 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet 01a10926bce14cf1bd12f21287e08c89 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:39229: session id = 533908e517b24f77b2af8b5863e4ae5a-01a10926bce14cf1bd12f21287e08c89
I20240501 19:56:36.732069 13669 consensus_queue.cc:237] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:36.733040 11400 tablet_copy_source_session.cc:205] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:36.733981 13932 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.734375 13800 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.734422 15427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a10926bce14cf1bd12f21287e08c89. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 19:56:36.734760 13603 consensus_peers.cc:485] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): Couldn't send request to peer 08711534813d47ba85aaec0771ed5d0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcb07fd6b982437195ff42e5bf0c48be. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.735105 14618 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:36.736294 13669 raft_consensus.cc:1203] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:36.736403 15427 tablet_copy_client.cc:783] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:36.736547 15427 tablet_copy_client.cc:647] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:36.738154 14618 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:36.739189 15427 tablet_copy_client.cc:523] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:36.740279 15481 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:36.740571 15427 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:36.740765 13932 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:36.743873 13800 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:36.744248 11106 catalog_manager.cc:5484] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:36.757962 14794 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.760668 14794 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.761809 11511 consensus_queue.cc:237] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:36.763569 11245 raft_consensus.cc:1261] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:36.763859 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c86f45702e94c0e898e4e81148f2098. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.764303 11377 raft_consensus.cc:1261] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.764724 14391 consensus_queue.cc:1035] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.765180 14391 consensus_queue.cc:1035] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.768662 14391 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.770511 11377 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.771497 11245 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.771677 11105 catalog_manager.cc:5484] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.773895 15427 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:36.774179 15427 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:36.774335 15427 ts_tablet_manager.cc:1367] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.000s
I20240501 19:56:36.774816 15427 raft_consensus.cc:348] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:36.774955 15427 raft_consensus.cc:729] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:36.775321 15427 consensus_queue.cc:260] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:36.775907 15427 ts_tablet_manager.cc:1398] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:36.777112 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-01a10926bce14cf1bd12f21287e08c89 received from {username='slave'} at 127.5.40.8:39229
I20240501 19:56:36.777186 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-01a10926bce14cf1bd12f21287e08c89 on tablet 01a10926bce14cf1bd12f21287e08c89 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:36.784453 11642 consensus_queue.cc:237] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:36.785871 13669 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:36.785907 11511 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:36.785954 11577 consensus_peers.cc:485] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e307d45fb57f444ca09d94d59b9bcd64. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.786255 14505 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:36.786509 14191 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:36.786969 11105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 19:56:36.787103 11105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 19:56:36.788471 14505 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.789309 11511 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.789780 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet b40fd5ed372743a1bb0680bd7de634fb
I20240501 19:56:36.789992 13669 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:36.790717 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet d9e0fc705bd4427ab966ff1edfae278b
I20240501 19:56:36.790961 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 1b74fc0672f34efaaee13512df8577ac
I20240501 19:56:36.791379 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet b40fd5ed372743a1bb0680bd7de634fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:36.791627 11100 catalog_manager.cc:5484] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 6 to 7, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.791908 14261 tablet_replica.cc:330] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:36.792054 14261 raft_consensus.cc:2229] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.792068 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b74fc0672f34efaaee13512df8577ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:36.792223 14261 raft_consensus.cc:2258] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.792227 14258 tablet_replica.cc:330] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:36.792315 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e0fc705bd4427ab966ff1edfae278b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:36.792362 14258 raft_consensus.cc:2229] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:36.792548 14258 raft_consensus.cc:2258] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.792771 14261 ts_tablet_manager.cc:1875] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.793057 14258 ts_tablet_manager.cc:1875] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.794723 14261 ts_tablet_manager.cc:1888] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.794799 14261 log.cc:1196] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/b40fd5ed372743a1bb0680bd7de634fb
I20240501 19:56:36.795022 14261 ts_tablet_manager.cc:1909] T b40fd5ed372743a1bb0680bd7de634fb P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:36.795140 14258 ts_tablet_manager.cc:1888] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.795204 14258 log.cc:1196] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/1b74fc0672f34efaaee13512df8577ac
I20240501 19:56:36.795485 14258 ts_tablet_manager.cc:1909] T 1b74fc0672f34efaaee13512df8577ac P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:36.795799 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e0fc705bd4427ab966ff1edfae278b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:36.795985 14258 tablet_replica.cc:330] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:36.796635 14258 raft_consensus.cc:2229] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.796813 14258 raft_consensus.cc:2258] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.797077 11228 tablet_service.cc:1514] Processing DeleteTablet for tablet b40fd5ed372743a1bb0680bd7de634fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:36.797230 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e0fc705bd4427ab966ff1edfae278b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:36.797363 15095 tablet_replica.cc:330] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:36.797492 15095 raft_consensus.cc:2229] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.797541 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b74fc0672f34efaaee13512df8577ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:36.797641 15095 raft_consensus.cc:2258] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.797799 11490 tablet_service.cc:1514] Processing DeleteTablet for tablet b40fd5ed372743a1bb0680bd7de634fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:36.798128 14258 ts_tablet_manager.cc:1875] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.798138 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b74fc0672f34efaaee13512df8577ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:36 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:36.798144 15095 ts_tablet_manager.cc:1875] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.798271 14711 tablet_replica.cc:330] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:36.798399 14711 raft_consensus.cc:2229] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:36.798588 14711 raft_consensus.cc:2258] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.799000 14711 ts_tablet_manager.cc:1875] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.800063 14258 ts_tablet_manager.cc:1888] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.800124 14258 log.cc:1196] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/d9e0fc705bd4427ab966ff1edfae278b
I20240501 19:56:36.800132 15095 ts_tablet_manager.cc:1888] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.800187 15095 log.cc:1196] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/b40fd5ed372743a1bb0680bd7de634fb
I20240501 19:56:36.800359 14258 ts_tablet_manager.cc:1909] T d9e0fc705bd4427ab966ff1edfae278b P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:36.800407 15095 ts_tablet_manager.cc:1909] T b40fd5ed372743a1bb0680bd7de634fb P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:36.800967 14711 ts_tablet_manager.cc:1888] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.801023 14711 log.cc:1196] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/b40fd5ed372743a1bb0680bd7de634fb
I20240501 19:56:36.801203 14711 ts_tablet_manager.cc:1909] T b40fd5ed372743a1bb0680bd7de634fb P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:36.801658 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 1b74fc0672f34efaaee13512df8577ac (table rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]) successfully deleted
I20240501 19:56:36.802204 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet d9e0fc705bd4427ab966ff1edfae278b (table rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]) successfully deleted
I20240501 19:56:36.802731 15507 consensus_queue.cc:1035] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.802930 14261 tablet_replica.cc:330] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:36.803085 14261 raft_consensus.cc:2229] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.803222 14261 raft_consensus.cc:2258] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.803282 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet b40fd5ed372743a1bb0680bd7de634fb (table rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]) successfully deleted
I20240501 19:56:36.803624 14711 tablet_replica.cc:330] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:36.803755 14711 raft_consensus.cc:2229] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.803875 14711 raft_consensus.cc:2258] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.804082 15095 tablet_replica.cc:330] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:36.804208 15095 raft_consensus.cc:2229] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:36.804332 14711 ts_tablet_manager.cc:1875] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.804384 15095 raft_consensus.cc:2258] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.804622 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet b40fd5ed372743a1bb0680bd7de634fb (table rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]) successfully deleted
I20240501 19:56:36.804853 15095 ts_tablet_manager.cc:1875] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.805151 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet b40fd5ed372743a1bb0680bd7de634fb (table rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]) successfully deleted
I20240501 19:56:36.805737 14261 ts_tablet_manager.cc:1875] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.806829 15095 ts_tablet_manager.cc:1888] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.806901 15095 log.cc:1196] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/d9e0fc705bd4427ab966ff1edfae278b
I20240501 19:56:36.807142 15095 ts_tablet_manager.cc:1909] T d9e0fc705bd4427ab966ff1edfae278b P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:36.807803 14935 consensus_queue.cc:1035] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.807997 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet d9e0fc705bd4427ab966ff1edfae278b (table rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]) successfully deleted
I20240501 19:56:36.808794 15095 tablet_replica.cc:330] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:36.808915 15095 raft_consensus.cc:2229] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:36.809080 15095 raft_consensus.cc:2258] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:36.809568 15095 ts_tablet_manager.cc:1875] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:36.810439 14711 ts_tablet_manager.cc:1888] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.810520 14711 log.cc:1196] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/1b74fc0672f34efaaee13512df8577ac
I20240501 19:56:36.810846 14261 ts_tablet_manager.cc:1888] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.810926 14261 log.cc:1196] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/d9e0fc705bd4427ab966ff1edfae278b
I20240501 19:56:36.811153 14261 ts_tablet_manager.cc:1909] T d9e0fc705bd4427ab966ff1edfae278b P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:36.811461 14711 ts_tablet_manager.cc:1909] T 1b74fc0672f34efaaee13512df8577ac P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:36.812153 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 1b74fc0672f34efaaee13512df8577ac (table rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]) successfully deleted
I20240501 19:56:36.812284 15095 ts_tablet_manager.cc:1888] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:36.812394 15095 log.cc:1196] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/1b74fc0672f34efaaee13512df8577ac
I20240501 19:56:36.812759 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet d9e0fc705bd4427ab966ff1edfae278b (table rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]) successfully deleted
I20240501 19:56:36.812805 15095 ts_tablet_manager.cc:1909] T 1b74fc0672f34efaaee13512df8577ac P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:36.813338 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 1b74fc0672f34efaaee13512df8577ac (table rebalance_test_table_12 [id=43b1ee8fc45e45ffad96ea800b93e606]) successfully deleted
I20240501 19:56:36.818495 13800 consensus_queue.cc:237] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
W20240501 19:56:36.819959 13734 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 -> Peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): Couldn't send request to peer 10ee60ead7b94c93ab749ec16a04b0dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8078e0cbffa943c98584224d05b0ece0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.820161 13669 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.820212 11642 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.820631 15507 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:36.820817 14935 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:36.822540 14935 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:36.822921 13669 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:36.823642 11642 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:36.825604 11105 catalog_manager.cc:5484] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.829178 11100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:36.832986 14063 raft_consensus.cc:1261] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.833194 13932 raft_consensus.cc:1261] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.833426 15481 consensus_queue.cc:1035] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.833910 15481 consensus_queue.cc:1035] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:36.836697 11106 catalog_manager.cc:5576] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789) does not have the latest schema for tablet 87475cc3ef6b4831889cce01290a70ce (table rebalancer_dynamic_table [id=63f9f6bda41a42779756e2f9fa74e4d5]). Expected version 1 got 0
I20240501 19:56:36.837106 13800 consensus_queue.cc:237] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:36.837359 15535 tablet.cc:1722] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:36.837690 15535 tablet.cc:1620] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:36.841305 15536 tablet.cc:1722] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:36.841634 15536 tablet.cc:1620] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:36.843964 14062 raft_consensus.cc:1261] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:36.843998 13669 raft_consensus.cc:1261] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:36.844194 13734 consensus_peers.cc:485] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fc692e320d94825964040105a3175cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.844408 15507 consensus_queue.cc:1035] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:36.844635 14935 consensus_queue.cc:1035] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:36.844866 11100 catalog_manager.cc:5576] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501) does not have the latest schema for tablet 87475cc3ef6b4831889cce01290a70ce (table rebalancer_dynamic_table [id=63f9f6bda41a42779756e2f9fa74e4d5]). Expected version 1 got 0
I20240501 19:56:36.845548 11106 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet 87475cc3ef6b4831889cce01290a70ce (table rebalancer_dynamic_table [id=63f9f6bda41a42779756e2f9fa74e4d5]). Expected version 1 got 0
I20240501 19:56:36.847678 15507 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:36.848471 13932 raft_consensus.cc:1203] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:36.850134 13669 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:36.850898 14062 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:36.851215 11105 catalog_manager.cc:5484] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:36.851883 15524 mvcc.cc:205] Tried to move back new op lower bound from 7022974553423654912 to 7022974552948289536. Current Snapshot: MvccSnapshot[applied={T|T < 7022974553423654912}]
I20240501 19:56:36.852718 11100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=63f9f6bda41a42779756e2f9fa74e4d5] alter complete (version 1)
I20240501 19:56:36.854194 11106 catalog_manager.cc:5576] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789) does not have the latest schema for tablet 87475cc3ef6b4831889cce01290a70ce (table rebalancer_dynamic_table [id=63f9f6bda41a42779756e2f9fa74e4d5]). Expected version 1 got 0
I20240501 19:56:36.859659 15537 tablet.cc:1722] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:36.860483 15537 tablet.cc:1620] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:36.865346 13800 raft_consensus.cc:1203] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:36.867853 11511 consensus_queue.cc:237] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:36.869277 11245 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:36.869292 11377 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:36.869562 11448 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3b35dab7eec4b73b208f687dad46ba2. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:36.869791 13437 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.869989 15028 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:36.872138 15028 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.872538 11245 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.875161 11097 catalog_manager.cc:5484] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.878662 11377 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:36.897686 13669 raft_consensus.cc:1203] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:36.957712 11097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:36.964591 15535 tablet.cc:1722] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:36.964956 15535 tablet.cc:1620] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:36.966185 15537 tablet.cc:1722] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:36.966536 15537 tablet.cc:1620] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:36.967140 15536 tablet.cc:1722] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:36.967454 15536 tablet.cc:1620] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:36.971102 11097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=63f9f6bda41a42779756e2f9fa74e4d5] alter complete (version 2)
I20240501 19:56:36.988118 11097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:36.994401 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 66bdddd3550b4ae8a97535e4747a5234 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.994484 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 66bdddd3550b4ae8a97535e4747a5234 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.994711 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bdddd3550b4ae8a97535e4747a5234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.994711 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bdddd3550b4ae8a97535e4747a5234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.995059 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8fedf0883b4b5ebee7e028a94a503b (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.995293 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8fedf0883b4b5ebee7e028a94a503b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.995996 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 66bdddd3550b4ae8a97535e4747a5234 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.996271 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bdddd3550b4ae8a97535e4747a5234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.997046 14042 tablet_service.cc:1467] Processing CreateTablet for tablet af9c3a48f92e40f19d75f4151bb4c3e3 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.997267 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9c3a48f92e40f19d75f4151bb4c3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.997334 13912 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8fedf0883b4b5ebee7e028a94a503b (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:36.997573 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8fedf0883b4b5ebee7e028a94a503b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:36.999049 15643 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:36.999267 15641 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:36.999959 15643 tablet_bootstrap.cc:654] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.000125 15641 tablet_bootstrap.cc:654] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.000177 13780 tablet_service.cc:1467] Processing CreateTablet for tablet af9c3a48f92e40f19d75f4151bb4c3e3 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.000424 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9c3a48f92e40f19d75f4151bb4c3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.000401 11227 tablet_service.cc:1467] Processing CreateTablet for tablet af9c3a48f92e40f19d75f4151bb4c3e3 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.000624 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9c3a48f92e40f19d75f4151bb4c3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.000720 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8fedf0883b4b5ebee7e028a94a503b (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.000897 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8fedf0883b4b5ebee7e028a94a503b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.003298 15642 tablet_bootstrap.cc:492] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:37.003468 15640 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:37.003985 15643 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:37.004070 15643 ts_tablet_manager.cc:1367] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:37.004465 15640 tablet_bootstrap.cc:654] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.004470 15642 tablet_bootstrap.cc:654] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.004462 15643 raft_consensus.cc:348] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.004555 15643 raft_consensus.cc:374] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.004612 15643 raft_consensus.cc:729] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.004746 15643 consensus_queue.cc:260] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.004895 15644 tablet_bootstrap.cc:492] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:37.005530 15643 ts_tablet_manager.cc:1398] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.005703 15643 tablet_bootstrap.cc:492] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:37.006608 15643 tablet_bootstrap.cc:654] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.007108 15644 tablet_bootstrap.cc:654] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.008713 15642 tablet_bootstrap.cc:492] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:37.008808 15642 ts_tablet_manager.cc:1367] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:37.008955 15644 tablet_bootstrap.cc:492] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:37.009052 15644 ts_tablet_manager.cc:1367] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:56:37.009505 15644 raft_consensus.cc:348] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:37.009641 15644 raft_consensus.cc:374] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.009694 15644 raft_consensus.cc:729] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.009672 15642 raft_consensus.cc:348] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:37.009740 15641 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:37.009816 15642 raft_consensus.cc:374] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.009829 15641 ts_tablet_manager.cc:1367] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:37.009873 15642 raft_consensus.cc:729] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.009881 15643 tablet_bootstrap.cc:492] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:37.009845 15644 consensus_queue.cc:260] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:37.009951 15643 ts_tablet_manager.cc:1367] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:37.010015 15642 consensus_queue.cc:260] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:37.010308 15642 ts_tablet_manager.cc:1398] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.010255 15641 raft_consensus.cc:348] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.010358 15641 raft_consensus.cc:374] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.010389 15641 raft_consensus.cc:729] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.010344 15643 raft_consensus.cc:348] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.010448 15643 raft_consensus.cc:374] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.010478 15643 raft_consensus.cc:729] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.010500 15641 consensus_queue.cc:260] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.010769 15641 ts_tablet_manager.cc:1398] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:37.010744 15643 consensus_queue.cc:260] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.010911 15641 tablet_bootstrap.cc:492] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:37.011626 15643 ts_tablet_manager.cc:1398] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.011709 15641 tablet_bootstrap.cc:654] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.011963 15644 ts_tablet_manager.cc:1398] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:37.015405 15641 tablet_bootstrap.cc:492] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:37.015559 15641 ts_tablet_manager.cc:1367] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:37.016045 15641 raft_consensus.cc:348] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.016162 15641 raft_consensus.cc:374] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.017401 15640 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:37.017520 15640 ts_tablet_manager.cc:1367] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:56:37.017956 15640 raft_consensus.cc:348] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.018079 15640 raft_consensus.cc:374] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.018110 15640 raft_consensus.cc:729] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.018232 15640 consensus_queue.cc:260] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.018505 15640 ts_tablet_manager.cc:1398] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:37.018682 15640 tablet_bootstrap.cc:492] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:37.019630 15640 tablet_bootstrap.cc:654] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.019891 15641 raft_consensus.cc:729] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.020081 15641 consensus_queue.cc:260] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.020390 15641 ts_tablet_manager.cc:1398] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 19:56:37.020733 15640 tablet_bootstrap.cc:492] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:37.020813 15640 ts_tablet_manager.cc:1367] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.021029 15641 tablet_bootstrap.cc:492] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:37.021346 15640 raft_consensus.cc:348] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.021522 15640 raft_consensus.cc:374] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.021589 15640 raft_consensus.cc:729] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.021732 15640 consensus_queue.cc:260] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.022085 15640 ts_tablet_manager.cc:1398] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.022063 15641 tablet_bootstrap.cc:654] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.023144 15641 tablet_bootstrap.cc:492] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:37.023224 15641 ts_tablet_manager.cc:1367] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:37.024250 15641 raft_consensus.cc:348] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:37.024426 15641 raft_consensus.cc:374] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.024511 15641 raft_consensus.cc:729] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.024684 15641 consensus_queue.cc:260] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:37.025038 15641 ts_tablet_manager.cc:1398] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.067265 15203 consensus_queue.cc:1035] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.073855 15203 consensus_queue.cc:1035] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.076195 11097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:37.082484 15535 tablet.cc:1722] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:37.083026 15537 tablet.cc:1722] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:37.084089 15536 tablet.cc:1722] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:37.088167 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=63f9f6bda41a42779756e2f9fa74e4d5] alter complete (version 3)
I20240501 19:56:37.091387 14062 raft_consensus.cc:1203] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:37.094633 15694 ts_tablet_manager.cc:902] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:37.095209 15694 tablet_copy_client.cc:308] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:42965
I20240501 19:56:37.095732 11269 tablet_copy_service.cc:140] P 10ee60ead7b94c93ab749ec16a04b0dc: Received BeginTabletCopySession request for tablet e96e8e7c66584b1198bea464e5a1a4d8 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:55515)
I20240501 19:56:37.095846 11269 tablet_copy_service.cc:161] P 10ee60ead7b94c93ab749ec16a04b0dc: Beginning new tablet copy session on tablet e96e8e7c66584b1198bea464e5a1a4d8 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:55515: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-e96e8e7c66584b1198bea464e5a1a4d8
I20240501 19:56:37.096789 11269 tablet_copy_source_session.cc:205] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.097698 15694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96e8e7c66584b1198bea464e5a1a4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.099959 15694 tablet_copy_client.cc:783] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.100106 15694 tablet_copy_client.cc:647] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.103212 15694 tablet_copy_client.cc:523] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.104396 15694 tablet_bootstrap.cc:492] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:37.107352 15692 raft_consensus.cc:482] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.107581 15692 raft_consensus.cc:504] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.108489 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bdddd3550b4ae8a97535e4747a5234" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:37.108525 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bdddd3550b4ae8a97535e4747a5234" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:37.108606 13800 raft_consensus.cc:2454] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:37.108678 11245 raft_consensus.cc:2454] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:37.109019 13999 leader_election.cc:302] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:37.109175 15203 raft_consensus.cc:2781] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.109212 15203 raft_consensus.cc:482] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.109242 15203 raft_consensus.cc:3037] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.109285 15692 leader_election.cc:288] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:37.110193 15203 raft_consensus.cc:504] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.110525 15203 leader_election.cc:288] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:37.111200 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bdddd3550b4ae8a97535e4747a5234" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:37.111307 11247 raft_consensus.cc:3037] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.112362 11247 raft_consensus.cc:2454] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:37.112686 13996 leader_election.cc:302] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 533908e517b24f77b2af8b5863e4ae5a; no voters:
I20240501 19:56:37.113142 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bdddd3550b4ae8a97535e4747a5234" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:37.113245 13800 raft_consensus.cc:3037] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.114383 13800 raft_consensus.cc:2454] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:37.115259 15203 raft_consensus.cc:2781] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.115345 15203 raft_consensus.cc:686] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:37.115497 15203 consensus_queue.cc:237] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.119029 11099 catalog_manager.cc:5484] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:37.132011 15694 tablet_bootstrap.cc:492] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.132289 15694 tablet_bootstrap.cc:492] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:37.132462 15694 ts_tablet_manager.cc:1367] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.000s
I20240501 19:56:37.132864 15694 raft_consensus.cc:348] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:37.133000 15694 raft_consensus.cc:729] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:37.133111 15694 consensus_queue.cc:260] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:37.133487 15694 ts_tablet_manager.cc:1398] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:37.134125 11269 tablet_copy_service.cc:342] P 10ee60ead7b94c93ab749ec16a04b0dc: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-e96e8e7c66584b1198bea464e5a1a4d8 received from {username='slave'} at 127.5.40.6:55515
I20240501 19:56:37.134253 11269 tablet_copy_service.cc:433] P 10ee60ead7b94c93ab749ec16a04b0dc: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-e96e8e7c66584b1198bea464e5a1a4d8 on tablet e96e8e7c66584b1198bea464e5a1a4d8 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:37.192956 11104 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:37.193141 11104 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:37.195521 11104 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 87475cc3ef6b4831889cce01290a70ce
I20240501 19:56:37.196321 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet 87475cc3ef6b4831889cce01290a70ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:37.196377 13650 tablet_service.cc:1514] Processing DeleteTablet for tablet 87475cc3ef6b4831889cce01290a70ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:37.196630 15701 tablet_replica.cc:330] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:37.196682 14042 tablet_service.cc:1514] Processing DeleteTablet for tablet 87475cc3ef6b4831889cce01290a70ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:37.196759 15701 raft_consensus.cc:2229] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.196862 15701 raft_consensus.cc:2258] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.197300 15702 tablet_replica.cc:330] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:37.197476 15702 raft_consensus.cc:2229] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:37.197510 15701 ts_tablet_manager.cc:1875] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.197683 15702 raft_consensus.cc:2258] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.198199 15702 ts_tablet_manager.cc:1875] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.199590 15703 tablet_replica.cc:330] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:37.199628 15701 ts_tablet_manager.cc:1888] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:37.199769 15701 log.cc:1196] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/87475cc3ef6b4831889cce01290a70ce
I20240501 19:56:37.199970 15703 raft_consensus.cc:2229] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.200083 15701 ts_tablet_manager.cc:1909] T 87475cc3ef6b4831889cce01290a70ce P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:37.200167 15702 ts_tablet_manager.cc:1888] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:37.200229 15703 raft_consensus.cc:2258] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.200286 15702 log.cc:1196] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/87475cc3ef6b4831889cce01290a70ce
I20240501 19:56:37.200616 15702 ts_tablet_manager.cc:1909] T 87475cc3ef6b4831889cce01290a70ce P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting consensus metadata
I20240501 19:56:37.201002 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet 87475cc3ef6b4831889cce01290a70ce (table rebalancer_dynamic_table_new_name [id=63f9f6bda41a42779756e2f9fa74e4d5]) successfully deleted
I20240501 19:56:37.201157 15703 ts_tablet_manager.cc:1875] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.201339 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet 87475cc3ef6b4831889cce01290a70ce (table rebalancer_dynamic_table_new_name [id=63f9f6bda41a42779756e2f9fa74e4d5]) successfully deleted
I20240501 19:56:37.202766 15703 ts_tablet_manager.cc:1888] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:37.202838 15703 log.cc:1196] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/87475cc3ef6b4831889cce01290a70ce
I20240501 19:56:37.203065 15703 ts_tablet_manager.cc:1909] T 87475cc3ef6b4831889cce01290a70ce P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:37.203980 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet 87475cc3ef6b4831889cce01290a70ce (table rebalancer_dynamic_table_new_name [id=63f9f6bda41a42779756e2f9fa74e4d5]) successfully deleted
I20240501 19:56:37.211175 14062 raft_consensus.cc:1203] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:37.235935 15427 ts_tablet_manager.cc:902] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:37.236501 15427 tablet_copy_client.cc:308] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:37.237241 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet e307d45fb57f444ca09d94d59b9bcd64 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:44731)
I20240501 19:56:37.237361 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet e307d45fb57f444ca09d94d59b9bcd64 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:44731: session id = 533908e517b24f77b2af8b5863e4ae5a-e307d45fb57f444ca09d94d59b9bcd64
I20240501 19:56:37.238412 11664 tablet_copy_source_session.cc:205] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.239470 15427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e307d45fb57f444ca09d94d59b9bcd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.241566 15427 tablet_copy_client.cc:783] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.241715 15427 tablet_copy_client.cc:647] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.242839 15427 tablet_copy_client.cc:523] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.243912 15427 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:37.245952 15708 ts_tablet_manager.cc:902] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:37.246496 15708 tablet_copy_client.cc:308] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:37.246968 11400 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet 169db26dfa3a4ec991c425c870e669eb from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:36785)
I20240501 19:56:37.247066 11400 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet 169db26dfa3a4ec991c425c870e669eb from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:36785: session id = 8125d8ad6d0d44caae9c5c90c90df96b-169db26dfa3a4ec991c425c870e669eb
I20240501 19:56:37.247853 11400 tablet_copy_source_session.cc:205] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.248641 15708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169db26dfa3a4ec991c425c870e669eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.250905 15708 tablet_copy_client.cc:783] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.251060 15708 tablet_copy_client.cc:647] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.252280 15708 tablet_copy_client.cc:523] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.253384 15708 tablet_bootstrap.cc:492] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:37.262583 15427 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.262831 15427 tablet_bootstrap.cc:492] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:37.263005 15427 ts_tablet_manager.cc:1367] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.002s
I20240501 19:56:37.263479 15427 raft_consensus.cc:348] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:37.263597 15427 raft_consensus.cc:729] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:37.263681 15427 consensus_queue.cc:260] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:37.263967 15427 ts_tablet_manager.cc:1398] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:37.264840 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-e307d45fb57f444ca09d94d59b9bcd64 received from {username='slave'} at 127.5.40.8:44731
I20240501 19:56:37.264926 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-e307d45fb57f444ca09d94d59b9bcd64 on tablet e307d45fb57f444ca09d94d59b9bcd64 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:37.274582 15708 tablet_bootstrap.cc:492] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.274871 15708 tablet_bootstrap.cc:492] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:37.275035 15708 ts_tablet_manager.cc:1367] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.000s
I20240501 19:56:37.275421 15708 raft_consensus.cc:348] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:37.275554 15715 ts_tablet_manager.cc:902] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:37.275563 15708 raft_consensus.cc:729] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:37.275660 15708 consensus_queue.cc:260] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:37.275930 15708 ts_tablet_manager.cc:1398] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240501 19:56:37.276072 15715 tablet_copy_client.cc:308] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:42965
I20240501 19:56:37.276546 11269 tablet_copy_service.cc:140] P 10ee60ead7b94c93ab749ec16a04b0dc: Received BeginTabletCopySession request for tablet 8f3176bda2ad4b9986cda48e4a03ce49 from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:42125)
I20240501 19:56:37.276643 11269 tablet_copy_service.cc:161] P 10ee60ead7b94c93ab749ec16a04b0dc: Beginning new tablet copy session on tablet 8f3176bda2ad4b9986cda48e4a03ce49 from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:42125: session id = 6921cf2e41f64d66842885512da8de6c-8f3176bda2ad4b9986cda48e4a03ce49
I20240501 19:56:37.276681 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-169db26dfa3a4ec991c425c870e669eb received from {username='slave'} at 127.5.40.5:36785
I20240501 19:56:37.276769 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-169db26dfa3a4ec991c425c870e669eb on tablet 169db26dfa3a4ec991c425c870e669eb with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:37.277341 11269 tablet_copy_source_session.cc:205] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.277957 15094 raft_consensus.cc:1050] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:37.278344 15094 consensus_queue.cc:237] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:37.278614 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3176bda2ad4b9986cda48e4a03ce49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.279764 11508 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.279925 11642 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.280220 15721 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.280418 14473 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.280774 15715 tablet_copy_client.cc:783] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.280926 15715 tablet_copy_client.cc:647] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.281001 13669 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.282326 15721 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.282985 15715 tablet_copy_client.cc:523] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.282948 14473 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.283702 11642 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.284088 15715 tablet_bootstrap.cc:492] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:37.284868 11508 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.286986 11104 catalog_manager.cc:5484] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:37.288977 13669 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.293673 11377 consensus_queue.cc:237] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:37.295401 13669 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.295627 11508 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.296029 15094 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.296669 15094 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.298022 11104 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:37.298200 11106 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 19:56:37.298307 11106 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 19:56:37.299283 15427 ts_tablet_manager.cc:902] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:37.299424 15094 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.299757 15427 tablet_copy_client.cc:308] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:42965
I20240501 19:56:37.300190 11269 tablet_copy_service.cc:140] P 10ee60ead7b94c93ab749ec16a04b0dc: Received BeginTabletCopySession request for tablet c31cf707a7e84f139fce27df22b376fd from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:50213)
I20240501 19:56:37.300287 11269 tablet_copy_service.cc:161] P 10ee60ead7b94c93ab749ec16a04b0dc: Beginning new tablet copy session on tablet c31cf707a7e84f139fce27df22b376fd from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:50213: session id = 533908e517b24f77b2af8b5863e4ae5a-c31cf707a7e84f139fce27df22b376fd
I20240501 19:56:37.301206 11269 tablet_copy_source_session.cc:205] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.301571 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6326c660e6cb4eb289634a68fc50c4af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:37.302286 15427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31cf707a7e84f139fce27df22b376fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.302735 11508 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.303959 11106 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 4f1f6382bcc84fcda444dcfae87b00c1
I20240501 19:56:37.304256 11106 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 85ba57eb6aae4f12b4a4641d633104a0
I20240501 19:56:37.304297 15427 tablet_copy_client.cc:783] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.304442 15427 tablet_copy_client.cc:647] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.304479 11106 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 582d72e630c84088a6e4a7309ae7e952
I20240501 19:56:37.304499 13669 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.305228 13650 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1f6382bcc84fcda444dcfae87b00c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:37.305259 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ba57eb6aae4f12b4a4641d633104a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:37.305379 15702 tablet_replica.cc:330] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:37.305426 15701 tablet_replica.cc:330] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:37.305500 15702 raft_consensus.cc:2229] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.305531 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1f6382bcc84fcda444dcfae87b00c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:37.305552 15701 raft_consensus.cc:2229] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.305624 15702 raft_consensus.cc:2258] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.305622 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ba57eb6aae4f12b4a4641d633104a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:37.305683 15701 raft_consensus.cc:2258] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.305953 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 582d72e630c84088a6e4a7309ae7e952 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:37.306303 15702 ts_tablet_manager.cc:1875] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.306455 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet 582d72e630c84088a6e4a7309ae7e952 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:37.306509 15701 ts_tablet_manager.cc:1875] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.306648 15731 tablet_replica.cc:330] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:37.306773 15731 raft_consensus.cc:2229] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:37.306874 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ba57eb6aae4f12b4a4641d633104a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:37.306963 15731 raft_consensus.cc:2258] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.307018 14711 tablet_replica.cc:330] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:37.307129 14711 raft_consensus.cc:2229] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:37.307312 14711 raft_consensus.cc:2258] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.307408 15731 ts_tablet_manager.cc:1875] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.308501 15701 ts_tablet_manager.cc:1888] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.308565 15701 log.cc:1196] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/85ba57eb6aae4f12b4a4641d633104a0
I20240501 19:56:37.308533 14391 raft_consensus.cc:1050] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:37.308640 15095 tablet_replica.cc:330] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:37.308796 15701 ts_tablet_manager.cc:1909] T 85ba57eb6aae4f12b4a4641d633104a0 P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:37.308795 15095 raft_consensus.cc:2229] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.308923 14391 consensus_queue.cc:237] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:37.308990 15095 raft_consensus.cc:2258] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.309487 15095 ts_tablet_manager.cc:1875] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.309713 15731 ts_tablet_manager.cc:1888] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.309775 15731 log.cc:1196] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/582d72e630c84088a6e4a7309ae7e952
I20240501 19:56:37.310010 15731 ts_tablet_manager.cc:1909] T 582d72e630c84088a6e4a7309ae7e952 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:37.310238 14711 ts_tablet_manager.cc:1875] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.310351 11642 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.310647 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 582d72e630c84088a6e4a7309ae7e952 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:37.310799 15028 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.311563 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1f6382bcc84fcda444dcfae87b00c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:37.311731 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 582d72e630c84088a6e4a7309ae7e952 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:37.311944 14261 tablet_replica.cc:330] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:37.312080 14261 raft_consensus.cc:2229] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:37.312117 14711 ts_tablet_manager.cc:1888] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.312177 14711 log.cc:1196] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/85ba57eb6aae4f12b4a4641d633104a0
I20240501 19:56:37.312273 14261 raft_consensus.cc:2258] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.312403 14711 ts_tablet_manager.cc:1909] T 85ba57eb6aae4f12b4a4641d633104a0 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:37.312772 14261 ts_tablet_manager.cc:1875] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.313313 13931 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.313917 15028 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.314170 14935 raft_consensus.cc:482] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.314257 14935 raft_consensus.cc:504] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.314592 14935 leader_election.cc:288] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:37.314859 14261 ts_tablet_manager.cc:1888] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.314924 14261 log.cc:1196] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/4f1f6382bcc84fcda444dcfae87b00c1
I20240501 19:56:37.314936 15732 tablet_replica.cc:330] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:37.315062 15732 raft_consensus.cc:2229] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.315152 14261 ts_tablet_manager.cc:1909] T 4f1f6382bcc84fcda444dcfae87b00c1 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:37.315183 15732 raft_consensus.cc:2258] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.315268 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9c3a48f92e40f19d75f4151bb4c3e3" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:37.315418 11247 raft_consensus.cc:2454] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:37.315483 15702 ts_tablet_manager.cc:1888] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.315426 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9c3a48f92e40f19d75f4151bb4c3e3" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:37.315563 15702 log.cc:1196] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/4f1f6382bcc84fcda444dcfae87b00c1
I20240501 19:56:37.315600 14063 raft_consensus.cc:2454] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:37.315692 15732 ts_tablet_manager.cc:1875] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.315750 13734 leader_election.cc:302] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:37.315824 15702 ts_tablet_manager.cc:1909] T 4f1f6382bcc84fcda444dcfae87b00c1 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting consensus metadata
I20240501 19:56:37.315898 14794 raft_consensus.cc:2781] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.315941 14794 raft_consensus.cc:482] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.315985 14794 raft_consensus.cc:3037] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.316893 14794 raft_consensus.cc:504] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.317251 14794 leader_election.cc:288] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:37.317252 11097 catalog_manager.cc:5484] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:37.317538 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet 4f1f6382bcc84fcda444dcfae87b00c1 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:37.317682 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 85ba57eb6aae4f12b4a4641d633104a0 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:37.318367 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9c3a48f92e40f19d75f4151bb4c3e3" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:37.318480 11247 raft_consensus.cc:3037] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.318650 11377 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.319201 15028 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.319422 11247 raft_consensus.cc:2454] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:37.319561 15095 ts_tablet_manager.cc:1888] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.319638 15095 log.cc:1196] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/4f1f6382bcc84fcda444dcfae87b00c1
I20240501 19:56:37.319900 15095 ts_tablet_manager.cc:1909] T 4f1f6382bcc84fcda444dcfae87b00c1 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:37.319871 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9c3a48f92e40f19d75f4151bb4c3e3" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:37.319994 14062 raft_consensus.cc:3037] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.320333 15715 tablet_bootstrap.cc:492] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.320479 15095 tablet_replica.cc:330] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:37.320595 15715 tablet_bootstrap.cc:492] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:37.320600 15095 raft_consensus.cc:2229] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.320745 15715 ts_tablet_manager.cc:1367] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.000s
I20240501 19:56:37.320756 15095 raft_consensus.cc:2258] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.320937 14062 raft_consensus.cc:2454] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:37.321264 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 4f1f6382bcc84fcda444dcfae87b00c1 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:37.321226 15715 raft_consensus.cc:348] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:37.321337 15715 raft_consensus.cc:729] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:37.321410 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet 85ba57eb6aae4f12b4a4641d633104a0 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:37.321427 15715 consensus_queue.cc:260] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:37.321650 13734 leader_election.cc:302] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:37.321738 15715 ts_tablet_manager.cc:1398] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:37.322106 14794 raft_consensus.cc:2781] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.322188 14794 raft_consensus.cc:686] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:37.322309 14794 consensus_queue.cc:237] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:37.322470 11269 tablet_copy_service.cc:342] P 10ee60ead7b94c93ab749ec16a04b0dc: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-8f3176bda2ad4b9986cda48e4a03ce49 received from {username='slave'} at 127.5.40.7:42125
I20240501 19:56:37.322535 11269 tablet_copy_service.cc:433] P 10ee60ead7b94c93ab749ec16a04b0dc: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-8f3176bda2ad4b9986cda48e4a03ce49 on tablet 8f3176bda2ad4b9986cda48e4a03ce49 with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:37.323117 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 6326c660e6cb4eb289634a68fc50c4af with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:37.323321 15731 tablet_replica.cc:330] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:37.323601 15731 raft_consensus.cc:2229] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.323779 15731 raft_consensus.cc:2258] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.324548 15731 ts_tablet_manager.cc:1875] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:37.324904 15095 ts_tablet_manager.cc:1875] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.326844 15731 ts_tablet_manager.cc:1888] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:37.326983 15731 log.cc:1196] T 6326c660e6cb4eb289634a68fc50c4af P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/6326c660e6cb4eb289634a68fc50c4af
I20240501 19:56:37.327528 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 2d88ad49e77b4070b7cf9ca888fac124 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3a4f3e40363545b6adf0d730d6511ce6]), partition=
I20240501 19:56:37.327528 13912 tablet_service.cc:1467] Processing CreateTablet for tablet 2d88ad49e77b4070b7cf9ca888fac124 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3a4f3e40363545b6adf0d730d6511ce6]), partition=
I20240501 19:56:37.327630 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 2d88ad49e77b4070b7cf9ca888fac124 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3a4f3e40363545b6adf0d730d6511ce6]), partition=
I20240501 19:56:37.327734 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d88ad49e77b4070b7cf9ca888fac124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.327739 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d88ad49e77b4070b7cf9ca888fac124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.328174 14261 tablet_replica.cc:330] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:37.328233 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 4f1f6382bcc84fcda444dcfae87b00c1 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:37.328331 14261 raft_consensus.cc:2229] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.328492 14261 raft_consensus.cc:2258] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.328974 14261 ts_tablet_manager.cc:1875] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.328843 11097 catalog_manager.cc:5484] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:37.329960 15644 tablet_bootstrap.cc:492] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:37.330166 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d88ad49e77b4070b7cf9ca888fac124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.330719 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 6326c660e6cb4eb289634a68fc50c4af (table rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]) successfully deleted
I20240501 19:56:37.330860 15644 tablet_bootstrap.cc:654] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.331593 14261 ts_tablet_manager.cc:1888] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.331662 14261 log.cc:1196] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/582d72e630c84088a6e4a7309ae7e952
I20240501 19:56:37.331905 14261 ts_tablet_manager.cc:1909] T 582d72e630c84088a6e4a7309ae7e952 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:37.332464 15644 tablet_bootstrap.cc:492] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:37.332562 15644 ts_tablet_manager.cc:1367] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:37.332989 15644 raft_consensus.cc:348] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.333114 15644 raft_consensus.cc:374] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.333143 15644 raft_consensus.cc:729] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.333299 15732 ts_tablet_manager.cc:1888] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.333276 15644 consensus_queue.cc:260] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.333360 15732 log.cc:1196] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/582d72e630c84088a6e4a7309ae7e952
I20240501 19:56:37.333541 15644 ts_tablet_manager.cc:1398] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:37.333618 15732 ts_tablet_manager.cc:1909] T 582d72e630c84088a6e4a7309ae7e952 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:37.333737 15640 tablet_bootstrap.cc:492] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:37.334278 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet 582d72e630c84088a6e4a7309ae7e952 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:37.334578 15640 tablet_bootstrap.cc:654] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.334599 15526 raft_consensus.cc:482] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.334677 15526 raft_consensus.cc:504] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.335103 15526 leader_election.cc:288] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:37.335680 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 582d72e630c84088a6e4a7309ae7e952 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:37.335837 13155 raft_consensus.cc:1050] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:37.336267 13155 consensus_queue.cc:237] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:37.336905 15738 ts_tablet_manager.cc:902] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: Initiating tablet copy from peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:37.337064 14934 raft_consensus.cc:1050] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:37.337342 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d88ad49e77b4070b7cf9ca888fac124" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:37.337456 14934 consensus_queue.cc:237] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:37.337631 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d88ad49e77b4070b7cf9ca888fac124" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:37.337685 15738 tablet_copy_client.cc:308] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.5:42501
W20240501 19:56:37.337880 13868 leader_election.cc:341] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Illegal state: must be running to vote when last-logged opid is not known
W20240501 19:56:37.338125 13866 leader_election.cc:341] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:37.338200 13866 leader_election.cc:302] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c; no voters: 8ad18cd54c6340fd92b874157875fa49, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:37.338410 15526 raft_consensus.cc:2726] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:37.339409 13800 raft_consensus.cc:1261] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.339836 11247 raft_consensus.cc:1261] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.340095 15748 ts_tablet_manager.cc:902] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:37.340124 14473 consensus_queue.cc:1035] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.340273 11643 raft_consensus.cc:1261] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.340838 14473 consensus_queue.cc:1035] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.340950 13690 tablet_copy_service.cc:140] P 8125d8ad6d0d44caae9c5c90c90df96b: Received BeginTabletCopySession request for tablet dcb07fd6b982437195ff42e5bf0c48be from peer 08711534813d47ba85aaec0771ed5d0f ({username='slave'} at 127.5.40.3:33027)
I20240501 19:56:37.341084 13690 tablet_copy_service.cc:161] P 8125d8ad6d0d44caae9c5c90c90df96b: Beginning new tablet copy session on tablet dcb07fd6b982437195ff42e5bf0c48be from peer 08711534813d47ba85aaec0771ed5d0f at {username='slave'} at 127.5.40.3:33027: session id = 08711534813d47ba85aaec0771ed5d0f-dcb07fd6b982437195ff42e5bf0c48be
I20240501 19:56:37.341436 15640 tablet_bootstrap.cc:492] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:37.341424 13437 consensus_queue.cc:1035] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.341463 11378 raft_consensus.cc:1261] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.341557 15640 ts_tablet_manager.cc:1367] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:37.341914 11643 raft_consensus.cc:1261] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.341979 15640 raft_consensus.cc:348] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.342097 15640 raft_consensus.cc:374] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.342132 15640 raft_consensus.cc:729] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.342125 13437 consensus_queue.cc:1035] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.342245 15640 consensus_queue.cc:260] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.342479 13690 tablet_copy_source_session.cc:205] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.342511 15640 ts_tablet_manager.cc:1398] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.342844 14473 consensus_queue.cc:1035] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.343636 13669 raft_consensus.cc:1261] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.345847 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb07fd6b982437195ff42e5bf0c48be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.345887 15095 ts_tablet_manager.cc:1888] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.345961 15095 log.cc:1196] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/85ba57eb6aae4f12b4a4641d633104a0
I20240501 19:56:37.346130 15028 consensus_queue.cc:1035] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.346222 15095 ts_tablet_manager.cc:1909] T 85ba57eb6aae4f12b4a4641d633104a0 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:37.347121 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 85ba57eb6aae4f12b4a4641d633104a0 (table rebalance_test_table_14 [id=c55bee38fca248c38af10542547d1126]) successfully deleted
I20240501 19:56:37.347019 14473 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:37.348145 15738 tablet_copy_client.cc:783] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.348140 11247 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:37.348310 15738 tablet_copy_client.cc:647] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.348793 15748 tablet_copy_client.cc:308] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.6:46331
I20240501 19:56:37.349273 14391 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.349541 13800 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:37.350181 11097 catalog_manager.cc:5484] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:37.352706 15739 tablet_bootstrap.cc:492] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:37.352686 13821 tablet_copy_service.cc:140] P 96fc18dbb5fd41afb9bebf805e8d33a7: Received BeginTabletCopySession request for tablet 1fc692e320d94825964040105a3175cd from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:60047)
I20240501 19:56:37.352798 13821 tablet_copy_service.cc:161] P 96fc18dbb5fd41afb9bebf805e8d33a7: Beginning new tablet copy session on tablet 1fc692e320d94825964040105a3175cd from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:60047: session id = 6921cf2e41f64d66842885512da8de6c-1fc692e320d94825964040105a3175cd
I20240501 19:56:37.352741 11642 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:37.353546 15739 tablet_bootstrap.cc:654] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.354111 13821 tablet_copy_source_session.cc:205] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.354660 11643 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.355262 15427 tablet_copy_client.cc:523] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.355286 15748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc692e320d94825964040105a3175cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.356491 13155 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:37.356673 15427 tablet_bootstrap.cc:492] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:37.358352 13931 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:37.360471 13669 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.360720 15739 tablet_bootstrap.cc:492] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:37.360859 15739 ts_tablet_manager.cc:1367] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:37.361598 11380 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:37.362161 11100 catalog_manager.cc:5484] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:37.362709 11106 catalog_manager.cc:5484] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:37.361313 15739 raft_consensus.cc:348] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.364293 15739 raft_consensus.cc:374] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.364382 15739 raft_consensus.cc:729] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.364557 15739 consensus_queue.cc:260] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.364974 15739 ts_tablet_manager.cc:1398] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:37.361599 11377 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.366571 11643 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:37.368613 11377 consensus_queue.cc:237] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:37.370008 13800 raft_consensus.cc:1261] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.370460 15094 consensus_queue.cc:1035] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.371192 11643 raft_consensus.cc:1261] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.372303 15094 consensus_queue.cc:1035] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.372925 15094 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:37.375099 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01ed8143e5454a4aaef3e142069f35a2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:37.375411 15694 ts_tablet_manager.cc:902] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:37.375304 13799 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:37.375875 11511 consensus_queue.cc:237] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:37.375967 15694 tablet_copy_client.cc:308] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:37.376401 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 2c86f45702e94c0e898e4e81148f2098 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:51789)
I20240501 19:56:37.376487 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 2c86f45702e94c0e898e4e81148f2098 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:51789: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-2c86f45702e94c0e898e4e81148f2098
I20240501 19:56:37.377048 11508 consensus_queue.cc:237] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:37.377408 11533 tablet_copy_source_session.cc:205] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.378036 11642 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.378327 13669 raft_consensus.cc:1261] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.378556 14391 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.378607 11097 catalog_manager.cc:5484] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:37.379019 14391 consensus_queue.cc:1035] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.380434 15694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c86f45702e94c0e898e4e81148f2098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.381042 13931 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.381620 15028 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.382316 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ed8143e5454a4aaef3e142069f35a2 with delete_type TABLET_DATA_TOMBSTONED (TS 10ee60ead7b94c93ab749ec16a04b0dc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:37.382710 15694 tablet_copy_client.cc:783] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.382860 15694 tablet_copy_client.cc:647] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.383124 11377 raft_consensus.cc:1261] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.383594 15095 tablet_replica.cc:330] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:37.383594 13155 consensus_queue.cc:1035] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.383744 15095 raft_consensus.cc:2229] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.383919 15095 raft_consensus.cc:2258] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.384527 11643 raft_consensus.cc:2932] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:37.384863 13155 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:37.386423 13931 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:37.387703 14391 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.388703 15738 tablet_copy_client.cc:523] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.389816 11642 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:37.390225 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe34db1ffe5f4509abb547f88c12a562 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:37.390302 15738 tablet_bootstrap.cc:492] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:37.391942 13669 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.395021 11105 catalog_manager.cc:5484] T fe34db1ffe5f4509abb547f88c12a562 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:37.395854 15095 ts_tablet_manager.cc:1875] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:37.397944 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet fe34db1ffe5f4509abb547f88c12a562 with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:37.398149 15731 tablet_replica.cc:330] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:37.398288 15731 raft_consensus.cc:2229] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.398453 15731 raft_consensus.cc:2258] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.398712 15095 ts_tablet_manager.cc:1888] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:37.398794 15095 log.cc:1196] T 01ed8143e5454a4aaef3e142069f35a2 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/01ed8143e5454a4aaef3e142069f35a2
I20240501 19:56:37.399153 15731 ts_tablet_manager.cc:1875] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:37.400116 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 01ed8143e5454a4aaef3e142069f35a2 (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:37.400836 11097 catalog_manager.cc:5484] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:37.401214 15694 tablet_copy_client.cc:523] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.401242 15731 ts_tablet_manager.cc:1888] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:37.401305 15731 log.cc:1196] T fe34db1ffe5f4509abb547f88c12a562 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/fe34db1ffe5f4509abb547f88c12a562
I20240501 19:56:37.401156 11377 raft_consensus.cc:2932] T fe34db1ffe5f4509abb547f88c12a562 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:37.402590 15694 tablet_bootstrap.cc:492] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:37.402832 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet fe34db1ffe5f4509abb547f88c12a562 (table rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]) successfully deleted
I20240501 19:56:37.404776 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00b7dd445e2846d5a0e1277e160d46bc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:37.408886 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b7dd445e2846d5a0e1277e160d46bc with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:37.409219 14261 tablet_replica.cc:330] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:37.409329 15748 tablet_copy_client.cc:783] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.409361 14261 raft_consensus.cc:2229] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.409487 15748 tablet_copy_client.cc:647] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.409512 14261 raft_consensus.cc:2258] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.410231 14261 ts_tablet_manager.cc:1875] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:37.410970 15748 tablet_copy_client.cc:523] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.411124 15738 tablet_bootstrap.cc:492] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.411365 15738 tablet_bootstrap.cc:492] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: Bootstrap complete.
I20240501 19:56:37.411984 15738 ts_tablet_manager.cc:1367] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.005s
I20240501 19:56:37.412230 15748 tablet_bootstrap.cc:492] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:37.412426 14261 ts_tablet_manager.cc:1888] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:37.412485 14261 log.cc:1196] T 00b7dd445e2846d5a0e1277e160d46bc P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/00b7dd445e2846d5a0e1277e160d46bc
I20240501 19:56:37.412438 15738 raft_consensus.cc:348] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:37.412550 15738 raft_consensus.cc:729] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: LEARNER
I20240501 19:56:37.412643 15738 consensus_queue.cc:260] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:37.412998 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 00b7dd445e2846d5a0e1277e160d46bc (table rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]) successfully deleted
I20240501 19:56:37.413262 15738 ts_tablet_manager.cc:1398] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:37.414021 13690 tablet_copy_service.cc:342] P 8125d8ad6d0d44caae9c5c90c90df96b: Request end of tablet copy session 08711534813d47ba85aaec0771ed5d0f-dcb07fd6b982437195ff42e5bf0c48be received from {username='slave'} at 127.5.40.3:33027
I20240501 19:56:37.414124 13690 tablet_copy_service.cc:433] P 8125d8ad6d0d44caae9c5c90c90df96b: ending tablet copy session 08711534813d47ba85aaec0771ed5d0f-dcb07fd6b982437195ff42e5bf0c48be on tablet dcb07fd6b982437195ff42e5bf0c48be with peer 08711534813d47ba85aaec0771ed5d0f
I20240501 19:56:37.415793 15748 tablet_bootstrap.cc:492] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.416582 15748 tablet_bootstrap.cc:492] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:37.416575 15427 tablet_bootstrap.cc:492] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.416795 15427 tablet_bootstrap.cc:492] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:37.416800 15748 ts_tablet_manager.cc:1367] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:37.417018 15427 ts_tablet_manager.cc:1367] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.060s user 0.026s sys 0.000s
I20240501 19:56:37.417243 15748 raft_consensus.cc:348] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:37.417469 15748 raft_consensus.cc:729] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:37.417532 15427 raft_consensus.cc:348] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:37.417600 15748 consensus_queue.cc:260] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:37.417645 15427 raft_consensus.cc:729] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:37.417734 15427 consensus_queue.cc:260] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:37.417912 15748 ts_tablet_manager.cc:1398] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.418478 13821 tablet_copy_service.cc:342] P 96fc18dbb5fd41afb9bebf805e8d33a7: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-1fc692e320d94825964040105a3175cd received from {username='slave'} at 127.5.40.7:60047
I20240501 19:56:37.418571 13821 tablet_copy_service.cc:433] P 96fc18dbb5fd41afb9bebf805e8d33a7: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-1fc692e320d94825964040105a3175cd on tablet 1fc692e320d94825964040105a3175cd with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:37.419751 15427 ts_tablet_manager.cc:1398] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:37.420738 11269 tablet_copy_service.cc:342] P 10ee60ead7b94c93ab749ec16a04b0dc: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-c31cf707a7e84f139fce27df22b376fd received from {username='slave'} at 127.5.40.8:50213
I20240501 19:56:37.420830 11269 tablet_copy_service.cc:433] P 10ee60ead7b94c93ab749ec16a04b0dc: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-c31cf707a7e84f139fce27df22b376fd on tablet c31cf707a7e84f139fce27df22b376fd with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:37.423614 15769 ts_tablet_manager.cc:902] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:37.424145 15769 tablet_copy_client.cc:308] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:37.424587 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet c3b35dab7eec4b73b208f687dad46ba2 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:51789)
I20240501 19:56:37.424702 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet c3b35dab7eec4b73b208f687dad46ba2 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:51789: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-c3b35dab7eec4b73b208f687dad46ba2
I20240501 19:56:37.425668 11533 tablet_copy_source_session.cc:205] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.425961 15774 ts_tablet_manager.cc:902] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: Initiating tablet copy from peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:37.426731 15769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b35dab7eec4b73b208f687dad46ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.426790 15774 tablet_copy_client.cc:308] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.6:46331
I20240501 19:56:37.427337 13821 tablet_copy_service.cc:140] P 96fc18dbb5fd41afb9bebf805e8d33a7: Received BeginTabletCopySession request for tablet 8078e0cbffa943c98584224d05b0ece0 from peer 10ee60ead7b94c93ab749ec16a04b0dc ({username='slave'} at 127.5.40.1:41511)
I20240501 19:56:37.427465 13821 tablet_copy_service.cc:161] P 96fc18dbb5fd41afb9bebf805e8d33a7: Beginning new tablet copy session on tablet 8078e0cbffa943c98584224d05b0ece0 from peer 10ee60ead7b94c93ab749ec16a04b0dc at {username='slave'} at 127.5.40.1:41511: session id = 10ee60ead7b94c93ab749ec16a04b0dc-8078e0cbffa943c98584224d05b0ece0
I20240501 19:56:37.428246 13821 tablet_copy_source_session.cc:205] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:37.428705 15769 tablet_copy_client.cc:783] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.428854 15769 tablet_copy_client.cc:647] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.429368 15774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8078e0cbffa943c98584224d05b0ece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.429850 15769 tablet_copy_client.cc:523] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.430820 15769 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:37.431892 15774 tablet_copy_client.cc:783] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:37.432109 15774 tablet_copy_client.cc:647] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:37.433825 15774 tablet_copy_client.cc:523] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.435314 15774 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:37.439615 15694 tablet_bootstrap.cc:492] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.439855 15694 tablet_bootstrap.cc:492] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:37.440037 15694 ts_tablet_manager.cc:1367] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.037s user 0.023s sys 0.000s
I20240501 19:56:37.440477 15526 raft_consensus.cc:482] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.440491 15694 raft_consensus.cc:348] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:37.440580 15526 raft_consensus.cc:504] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:37.440645 15694 raft_consensus.cc:729] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:37.440785 15694 consensus_queue.cc:260] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:37.440989 15526 leader_election.cc:288] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:37.441404 15774 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.441418 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8fedf0883b4b5ebee7e028a94a503b" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:37.441571 11377 raft_consensus.cc:2454] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:37.441691 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8fedf0883b4b5ebee7e028a94a503b" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:37.441805 11247 raft_consensus.cc:2454] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:37.441849 15774 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap complete.
I20240501 19:56:37.441987 15774 ts_tablet_manager.cc:1367] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 19:56:37.442179 13867 leader_election.cc:302] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:37.442363 15694 ts_tablet_manager.cc:1398] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.442423 15774 raft_consensus.cc:348] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:37.442534 15774 raft_consensus.cc:729] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: LEARNER
I20240501 19:56:37.442535 15526 raft_consensus.cc:2781] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.442598 15526 raft_consensus.cc:482] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.442638 15526 raft_consensus.cc:3037] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.442625 15774 consensus_queue.cc:260] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:37.442925 15774 ts_tablet_manager.cc:1398] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.442920 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-2c86f45702e94c0e898e4e81148f2098 received from {username='slave'} at 127.5.40.6:51789
I20240501 19:56:37.442998 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-2c86f45702e94c0e898e4e81148f2098 on tablet 2c86f45702e94c0e898e4e81148f2098 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:37.443399 13821 tablet_copy_service.cc:342] P 96fc18dbb5fd41afb9bebf805e8d33a7: Request end of tablet copy session 10ee60ead7b94c93ab749ec16a04b0dc-8078e0cbffa943c98584224d05b0ece0 received from {username='slave'} at 127.5.40.1:41511
I20240501 19:56:37.443506 13821 tablet_copy_service.cc:433] P 96fc18dbb5fd41afb9bebf805e8d33a7: ending tablet copy session 10ee60ead7b94c93ab749ec16a04b0dc-8078e0cbffa943c98584224d05b0ece0 on tablet 8078e0cbffa943c98584224d05b0ece0 with peer 10ee60ead7b94c93ab749ec16a04b0dc
I20240501 19:56:37.443719 15526 raft_consensus.cc:504] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:37.444097 15526 leader_election.cc:288] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:37.444334 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8fedf0883b4b5ebee7e028a94a503b" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:37.444428 11247 raft_consensus.cc:3037] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.445125 11247 raft_consensus.cc:2454] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:37.445416 13865 leader_election.cc:302] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:37.445611 15526 raft_consensus.cc:2781] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.445751 15526 raft_consensus.cc:686] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Becoming Leader. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Running, Role: LEADER
I20240501 19:56:37.445876 15526 consensus_queue.cc:237] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:37.447741 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8fedf0883b4b5ebee7e028a94a503b" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:37.447885 11377 raft_consensus.cc:3037] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.448305 11105 catalog_manager.cc:5484] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921cf2e41f64d66842885512da8de6c (127.5.40.7). New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:37.448887 11377 raft_consensus.cc:2454] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:37.462577 14794 raft_consensus.cc:482] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:37.462695 14794 raft_consensus.cc:504] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.463068 14794 leader_election.cc:288] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:37.463403 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d88ad49e77b4070b7cf9ca888fac124" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:37.463593 13931 raft_consensus.cc:2454] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:37.463588 11642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d88ad49e77b4070b7cf9ca888fac124" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:37.463737 11642 raft_consensus.cc:2454] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:37.464004 13734 leader_election.cc:302] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:37.464500 14794 raft_consensus.cc:2781] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:37.464582 14794 raft_consensus.cc:482] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:37.464630 14794 raft_consensus.cc:3037] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.465626 14794 raft_consensus.cc:504] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.465979 14794 leader_election.cc:288] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:37.466387 11642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d88ad49e77b4070b7cf9ca888fac124" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:37.466545 11642 raft_consensus.cc:3037] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.466545 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d88ad49e77b4070b7cf9ca888fac124" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:37.466658 13931 raft_consensus.cc:3037] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.467617 11642 raft_consensus.cc:2454] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:37.467753 13931 raft_consensus.cc:2454] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:37.467897 13735 leader_election.cc:302] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:37.468103 14794 raft_consensus.cc:2781] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.468195 14794 raft_consensus.cc:686] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:37.468327 14794 consensus_queue.cc:237] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:37.464489 15769 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.470537 15769 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:37.470769 15769 ts_tablet_manager.cc:1367] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.040s user 0.020s sys 0.008s
I20240501 19:56:37.471007 11100 catalog_manager.cc:5484] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:37.471264 15769 raft_consensus.cc:348] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:37.471367 15769 raft_consensus.cc:729] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:37.471486 15769 consensus_queue.cc:260] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:37.472144 15769 ts_tablet_manager.cc:1398] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:37.472600 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-c3b35dab7eec4b73b208f687dad46ba2 received from {username='slave'} at 127.5.40.6:51789
I20240501 19:56:37.472678 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-c3b35dab7eec4b73b208f687dad46ba2 on tablet c3b35dab7eec4b73b208f687dad46ba2 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:37.473459 11227 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:37.473528 11491 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 19:56:37.477780 13780 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:37.480005 14042 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:37.482796 11359 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:37.486809 13912 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:37.501157 13650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:37.502615 11624 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:37.535205 13799 raft_consensus.cc:1203] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:37.564934 11642 consensus_queue.cc:237] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:37.566404 11508 raft_consensus.cc:1261] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:37.566525 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1772073172764f8db95c597f8da14f12. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:37.566783 11247 raft_consensus.cc:1261] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:37.566809 15806 consensus_queue.cc:1035] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:37.567130 15081 consensus_queue.cc:1035] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:37.568609 15806 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:37.568969 11508 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:37.569859 11247 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:37.572870 11105 catalog_manager.cc:5484] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:37.575426 11100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:37.575822 15461 raft_consensus.cc:1050] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:37.576212 15461 consensus_queue.cc:237] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:37.577724 11642 raft_consensus.cc:1261] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.578331 11508 raft_consensus.cc:1261] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.578461 14062 raft_consensus.cc:1261] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.578819 15815 consensus_queue.cc:1035] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.579146 15815 consensus_queue.cc:1035] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.580371 15462 consensus_queue.cc:1035] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.581710 11643 raft_consensus.cc:1261] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.582448 13931 raft_consensus.cc:1261] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.582507 14794 consensus_queue.cc:1035] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.582928 14794 consensus_queue.cc:1035] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.584970 15815 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.586442 11097 catalog_manager.cc:5576] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041) does not have the latest schema for tablet 2d88ad49e77b4070b7cf9ca888fac124 (table rebalancer_dynamic_table [id=3a4f3e40363545b6adf0d730d6511ce6]). Expected version 1 got 0
I20240501 19:56:37.586638 11644 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.587051 14062 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.587296 11100 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet 2d88ad49e77b4070b7cf9ca888fac124 (table rebalancer_dynamic_table [id=3a4f3e40363545b6adf0d730d6511ce6]). Expected version 1 got 0
I20240501 19:56:37.589128 15824 tablet.cc:1722] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:37.590152 15824 tablet.cc:1620] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:37.590998 11100 catalog_manager.cc:5484] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.592293 11377 consensus_queue.cc:237] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:37.593755 11097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=3a4f3e40363545b6adf0d730d6511ce6] alter complete (version 1)
I20240501 19:56:37.594138 11644 raft_consensus.cc:1261] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:37.594339 11511 raft_consensus.cc:1261] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:37.594830 14934 consensus_queue.cc:1035] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:37.594966 15825 tablet.cc:1722] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:37.595263 15825 tablet.cc:1620] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: MemRowSet was empty: no flush needed.
I20240501 19:56:37.595993 14934 consensus_queue.cc:1035] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:37.596529 15827 tablet.cc:1722] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:37.597524 15827 tablet.cc:1620] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:37.597836 11511 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
W20240501 19:56:37.598343 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61cadb3ab334435eaebe99fc3c9e496b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:37.599207 11105 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet 2d88ad49e77b4070b7cf9ca888fac124 (table rebalancer_dynamic_table [id=3a4f3e40363545b6adf0d730d6511ce6]). Expected version 1 got 0
I20240501 19:56:37.599869 11247 consensus_queue.cc:237] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:37.601172 11644 raft_consensus.cc:1261] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.601645 15815 consensus_queue.cc:1035] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.601732 14934 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:37.602830 14062 raft_consensus.cc:1261] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.603300 15461 consensus_queue.cc:1035] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.604138 11511 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:37.605414 11644 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:37.605603 11100 catalog_manager.cc:5484] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:37.606793 15461 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.608120 14062 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.609519 11644 raft_consensus.cc:2932] T 1b57520d53734c318d603dd46e30b190 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.609870 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b57520d53734c318d603dd46e30b190 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:37.611527 11100 catalog_manager.cc:5484] T 1b57520d53734c318d603dd46e30b190 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:37.613885 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b57520d53734c318d603dd46e30b190 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:37.614089 14711 tablet_replica.cc:330] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:37.614233 14711 raft_consensus.cc:2229] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.614387 14711 raft_consensus.cc:2258] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.615085 14711 ts_tablet_manager.cc:1875] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:37.617192 14711 ts_tablet_manager.cc:1888] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:37.617287 14711 log.cc:1196] T 1b57520d53734c318d603dd46e30b190 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/1b57520d53734c318d603dd46e30b190
I20240501 19:56:37.617723 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 1b57520d53734c318d603dd46e30b190 (table rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]) successfully deleted
I20240501 19:56:37.640362 15203 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.645040 15203 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.708196 14062 raft_consensus.cc:1203] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:37.710122 11100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:37.716445 15824 tablet.cc:1722] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:37.716773 15824 tablet.cc:1620] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:37.716774 15825 tablet.cc:1722] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:37.717052 15825 tablet.cc:1620] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: MemRowSet was empty: no flush needed.
I20240501 19:56:37.718091 15827 tablet.cc:1722] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:37.718353 15827 tablet.cc:1620] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:37.722829 11100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=3a4f3e40363545b6adf0d730d6511ce6] alter complete (version 2)
I20240501 19:56:37.745266 13667 consensus_queue.cc:237] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:37.746654 14062 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.746778 11247 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:37.746872 13604 consensus_peers.cc:485] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): Couldn't send request to peer 8ad18cd54c6340fd92b874157875fa49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96511455168148b7b619670f635d3a3e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:37.747238 15930 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:37.747462 15931 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:37.748836 13667 raft_consensus.cc:1203] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:37.749485 15934 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:37.749855 11247 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:37.749895 14062 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:37.752779 11100 catalog_manager.cc:5484] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:37.756193 14062 raft_consensus.cc:1203] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:37.762094 11511 raft_consensus.cc:1203] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:37.764191 11508 consensus_queue.cc:237] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:37.765760 11642 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:37.766037 13667 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:37.766371 13437 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:37.766577 15765 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:37.768294 13437 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:37.769402 13667 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
W20240501 19:56:37.769601 11445 consensus_peers.cc:485] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e63472cfff449089d88d18b9e495912. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:37.771497 11104 catalog_manager.cc:5484] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 6 to 7, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:37.772347 11642 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:37.781455 11508 consensus_queue.cc:237] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:37.782851 11247 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:37.782944 13931 raft_consensus.cc:1261] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:37.783109 11445 consensus_peers.cc:485] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7073666bc3f4c0a8c1e003fb1ef4fef. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:37.783318 13155 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:37.783946 13155 consensus_queue.cc:1035] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:37.785035 15028 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:37.785455 13931 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:37.786201 11247 raft_consensus.cc:2932] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:37.787935 11105 catalog_manager.cc:5484] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 6 to 7, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:37.790196 14473 raft_consensus.cc:1050] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:37.790640 14473 consensus_queue.cc:237] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:37.791956 11508 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.792215 14062 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.792557 11642 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:37.792549 15721 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.792770 14934 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.793339 14934 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:37.795328 15721 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.796739 11642 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.798204 14062 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.798281 11100 catalog_manager.cc:5484] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:37.798705 11508 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.802377 11377 consensus_queue.cc:237] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:37.802377 13799 consensus_queue.cc:237] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:37.804060 13931 raft_consensus.cc:1261] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:37.804239 11247 raft_consensus.cc:1261] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:37.804513 13736 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 -> Peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): Couldn't send request to peer ef60ab93b49c4346af4f62d3b98558b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87aabbb9dc2440f28a3b6944dea38fe7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:37.804551 14794 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:37.804738 11642 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.804745 15507 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:37.805045 14062 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:37.805348 14473 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.806308 14473 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:37.806735 11105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 19:56:37.806854 11105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 19:56:37.807018 14794 raft_consensus.cc:2932] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:37.807263 14473 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.808461 14062 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.809098 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01a10926bce14cf1bd12f21287e08c89 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:37.809206 11642 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:37.810191 11104 catalog_manager.cc:5484] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:37.810463 11097 catalog_manager.cc:5484] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:37.811384 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 4dc0294d0dc5427e80411be5db9c95c1
I20240501 19:56:37.811699 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 64a03f21e05f4999843f48cb4ebdb63c
I20240501 19:56:37.811944 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet ea873ec8ced340b4ac082417c9b09aae
I20240501 19:56:37.811925 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc0294d0dc5427e80411be5db9c95c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:37.811937 11247 raft_consensus.cc:2932] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:37.812319 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a03f21e05f4999843f48cb4ebdb63c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:37.812644 15731 tablet_replica.cc:330] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:37.812767 15731 raft_consensus.cc:2229] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.812888 15731 raft_consensus.cc:2258] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.813306 14261 tablet_replica.cc:330] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:37.813318 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc0294d0dc5427e80411be5db9c95c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:37.813360 15731 ts_tablet_manager.cc:1875] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.813416 11490 tablet_service.cc:1514] Processing DeleteTablet for tablet ea873ec8ced340b4ac082417c9b09aae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:37.813432 14261 raft_consensus.cc:2229] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.813493 14711 tablet_replica.cc:330] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:37.813608 14711 raft_consensus.cc:2229] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:37.813611 14261 raft_consensus.cc:2258] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.813789 14711 raft_consensus.cc:2258] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.814054 14261 ts_tablet_manager.cc:1875] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.814285 14711 ts_tablet_manager.cc:1875] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.815488 15731 ts_tablet_manager.cc:1888] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:37.815553 15731 log.cc:1196] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/64a03f21e05f4999843f48cb4ebdb63c
I20240501 19:56:37.815743 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet ea873ec8ced340b4ac082417c9b09aae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:37.815790 15731 ts_tablet_manager.cc:1909] T 64a03f21e05f4999843f48cb4ebdb63c P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:37.816295 15095 tablet_replica.cc:330] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:37.816375 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet ea873ec8ced340b4ac082417c9b09aae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:37.816401 15095 raft_consensus.cc:2229] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.816566 15095 raft_consensus.cc:2258] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.816622 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a03f21e05f4999843f48cb4ebdb63c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:37.816663 14711 ts_tablet_manager.cc:1888] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.816715 14711 log.cc:1196] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/4dc0294d0dc5427e80411be5db9c95c1
I20240501 19:56:37.816936 14711 ts_tablet_manager.cc:1909] T 4dc0294d0dc5427e80411be5db9c95c1 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:37.817015 15095 ts_tablet_manager.cc:1875] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.817016 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a03f21e05f4999843f48cb4ebdb63c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:37.817248 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc0294d0dc5427e80411be5db9c95c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:37.817420 15732 tablet_replica.cc:330] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:37.817534 15732 raft_consensus.cc:2229] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.817651 15732 raft_consensus.cc:2258] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.817660 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 64a03f21e05f4999843f48cb4ebdb63c (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:37.817939 14711 tablet_replica.cc:330] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:37.818044 14711 raft_consensus.cc:2229] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:37.818042 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 4dc0294d0dc5427e80411be5db9c95c1 (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:37.818166 15732 ts_tablet_manager.cc:1875] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.818205 14711 raft_consensus.cc:2258] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.818619 14711 ts_tablet_manager.cc:1875] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.818804 15095 ts_tablet_manager.cc:1888] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.818851 15095 log.cc:1196] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/ea873ec8ced340b4ac082417c9b09aae
I20240501 19:56:37.819043 15095 ts_tablet_manager.cc:1909] T ea873ec8ced340b4ac082417c9b09aae P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:37.819630 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet ea873ec8ced340b4ac082417c9b09aae (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:37.820228 15732 ts_tablet_manager.cc:1888] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.820292 14711 ts_tablet_manager.cc:1888] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.820309 15732 log.cc:1196] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/4dc0294d0dc5427e80411be5db9c95c1
I20240501 19:56:37.820343 14711 log.cc:1196] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/ea873ec8ced340b4ac082417c9b09aae
I20240501 19:56:37.820559 14711 ts_tablet_manager.cc:1909] T ea873ec8ced340b4ac082417c9b09aae P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:37.820564 15732 ts_tablet_manager.cc:1909] T 4dc0294d0dc5427e80411be5db9c95c1 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:37.821180 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet ea873ec8ced340b4ac082417c9b09aae (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:37.821192 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet 4dc0294d0dc5427e80411be5db9c95c1 (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:37.821419 13931 raft_consensus.cc:2932] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:37.822419 11247 raft_consensus.cc:1203] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:37.823935 13931 raft_consensus.cc:1203] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:37.825803 11490 tablet_service.cc:1514] Processing DeleteTablet for tablet 01a10926bce14cf1bd12f21287e08c89 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:37.825819 14261 ts_tablet_manager.cc:1888] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.825887 14261 log.cc:1196] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/4dc0294d0dc5427e80411be5db9c95c1
I20240501 19:56:37.826123 14261 ts_tablet_manager.cc:1909] T 4dc0294d0dc5427e80411be5db9c95c1 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:37.826331 14711 tablet_replica.cc:330] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:37.826454 14711 raft_consensus.cc:2229] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.826555 11104 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:37.826642 14711 raft_consensus.cc:2258] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.826813 15964 tablet_replica.cc:330] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:37.826930 15964 raft_consensus.cc:2229] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.827044 15964 raft_consensus.cc:2258] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.827615 15964 ts_tablet_manager.cc:1875] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.829974 15964 ts_tablet_manager.cc:1888] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:37.830093 15964 log.cc:1196] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/ea873ec8ced340b4ac082417c9b09aae
I20240501 19:56:37.830458 14261 tablet_replica.cc:330] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:37.830637 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 4dc0294d0dc5427e80411be5db9c95c1 (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:37.830725 15964 ts_tablet_manager.cc:1909] T ea873ec8ced340b4ac082417c9b09aae P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:37.831357 14711 ts_tablet_manager.cc:1875] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:37.833128 15095 tablet_replica.cc:330] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:37.833292 15095 raft_consensus.cc:2229] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:37.833490 15095 raft_consensus.cc:2258] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.833539 15824 tablet.cc:1722] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:37.833953 15095 ts_tablet_manager.cc:1875] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.834034 15825 tablet.cc:1722] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:37.834424 14261 raft_consensus.cc:2229] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.834568 14261 raft_consensus.cc:2258] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.835018 14261 ts_tablet_manager.cc:1875] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.837505 14711 ts_tablet_manager.cc:1888] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:37.837765 14711 log.cc:1196] T 01a10926bce14cf1bd12f21287e08c89 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/01a10926bce14cf1bd12f21287e08c89
I20240501 19:56:37.838341 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 01a10926bce14cf1bd12f21287e08c89 (table rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]) successfully deleted
I20240501 19:56:37.838637 11097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=3a4f3e40363545b6adf0d730d6511ce6] alter complete (version 3)
I20240501 19:56:37.838829 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet ea873ec8ced340b4ac082417c9b09aae (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:37.839462 15827 tablet.cc:1722] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:37.841022 14261 ts_tablet_manager.cc:1888] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:37.841090 14261 log.cc:1196] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/64a03f21e05f4999843f48cb4ebdb63c
I20240501 19:56:37.841298 14261 ts_tablet_manager.cc:1909] T 64a03f21e05f4999843f48cb4ebdb63c P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:37.842178 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 64a03f21e05f4999843f48cb4ebdb63c (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:37.843324 15095 ts_tablet_manager.cc:1888] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:56:37.843407 15095 log.cc:1196] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/64a03f21e05f4999843f48cb4ebdb63c
I20240501 19:56:37.843665 15095 ts_tablet_manager.cc:1909] T 64a03f21e05f4999843f48cb4ebdb63c P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:37.844311 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 64a03f21e05f4999843f48cb4ebdb63c (table rebalance_test_table_16 [id=ac353816226142a38ca79ed5aee2a418]) successfully deleted
I20240501 19:56:37.847775 15526 consensus_queue.cc:1035] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.851502 15781 consensus_queue.cc:1035] T 2d8fedf0883b4b5ebee7e028a94a503b P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.853839 11377 consensus_queue.cc:237] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
W20240501 19:56:37.855580 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cebd15701ee44b11a2f8804740fb4299. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:37.855634 11642 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:37.856040 14934 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:37.856374 14062 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:37.856782 14934 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:37.857885 14934 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:37.860081 14062 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:37.860116 11642 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:37.860700 11104 catalog_manager.cc:5484] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:37.868256 13799 raft_consensus.cc:1203] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:37.898766 15507 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.904040 14935 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:37.908174 13931 raft_consensus.cc:1203] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:37.924038 13799 raft_consensus.cc:1203] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:37.942677 11105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:37.942873 11105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:37.945151 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 2d88ad49e77b4070b7cf9ca888fac124
I20240501 19:56:37.945719 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d88ad49e77b4070b7cf9ca888fac124 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:37.945873 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d88ad49e77b4070b7cf9ca888fac124 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:37.945947 15732 tablet_replica.cc:330] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:37.946087 15732 raft_consensus.cc:2229] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:37.946260 15731 tablet_replica.cc:330] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:37.946271 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d88ad49e77b4070b7cf9ca888fac124 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:37 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:37.946338 15732 raft_consensus.cc:2258] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.946408 15731 raft_consensus.cc:2229] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.946537 15731 raft_consensus.cc:2258] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.946540 15964 tablet_replica.cc:330] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:37.946671 15964 raft_consensus.cc:2229] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:37.946801 15964 raft_consensus.cc:2258] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:37.947391 15732 ts_tablet_manager.cc:1875] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.947683 15731 ts_tablet_manager.cc:1875] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.949687 15731 ts_tablet_manager.cc:1888] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:37.949687 15732 ts_tablet_manager.cc:1888] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:37.949751 15731 log.cc:1196] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/2d88ad49e77b4070b7cf9ca888fac124
I20240501 19:56:37.949751 15732 log.cc:1196] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/2d88ad49e77b4070b7cf9ca888fac124
I20240501 19:56:37.950003 15731 ts_tablet_manager.cc:1909] T 2d88ad49e77b4070b7cf9ca888fac124 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:37.950003 15732 ts_tablet_manager.cc:1909] T 2d88ad49e77b4070b7cf9ca888fac124 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:37.950688 15964 ts_tablet_manager.cc:1875] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:37.951021 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 2d88ad49e77b4070b7cf9ca888fac124 (table rebalancer_dynamic_table_new_name [id=3a4f3e40363545b6adf0d730d6511ce6]) successfully deleted
I20240501 19:56:37.952792 15964 ts_tablet_manager.cc:1888] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:37.952865 15964 log.cc:1196] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/2d88ad49e77b4070b7cf9ca888fac124
I20240501 19:56:37.953094 15964 ts_tablet_manager.cc:1909] T 2d88ad49e77b4070b7cf9ca888fac124 P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:37.955586 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet 2d88ad49e77b4070b7cf9ca888fac124 (table rebalancer_dynamic_table_new_name [id=3a4f3e40363545b6adf0d730d6511ce6]) successfully deleted
I20240501 19:56:37.956734 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet 2d88ad49e77b4070b7cf9ca888fac124 (table rebalancer_dynamic_table_new_name [id=3a4f3e40363545b6adf0d730d6511ce6]) successfully deleted
I20240501 19:56:37.990208 11105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:37.996690 11359 tablet_service.cc:1467] Processing CreateTablet for tablet f8b69cb1df2543f583eec2529a292897 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.997046 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b69cb1df2543f583eec2529a292897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.997258 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5f03eecaec4b929723c8f02a185606 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.997467 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5f03eecaec4b929723c8f02a185606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.997594 13650 tablet_service.cc:1467] Processing CreateTablet for tablet f8b69cb1df2543f583eec2529a292897 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.997895 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b69cb1df2543f583eec2529a292897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.998170 11490 tablet_service.cc:1467] Processing CreateTablet for tablet 811d018fde314227a378f2d530e0a924 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.998418 11490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811d018fde314227a378f2d530e0a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.999423 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5f03eecaec4b929723c8f02a185606 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:37.999701 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5f03eecaec4b929723c8f02a185606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.000218 14042 tablet_service.cc:1467] Processing CreateTablet for tablet f8b69cb1df2543f583eec2529a292897 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.000262 13648 tablet_service.cc:1467] Processing CreateTablet for tablet 811d018fde314227a378f2d530e0a924 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.000470 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b69cb1df2543f583eec2529a292897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.000473 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811d018fde314227a378f2d530e0a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.000787 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5f03eecaec4b929723c8f02a185606 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.000986 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5f03eecaec4b929723c8f02a185606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.001750 13912 tablet_service.cc:1467] Processing CreateTablet for tablet 811d018fde314227a378f2d530e0a924 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.001999 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811d018fde314227a378f2d530e0a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.003316 16073 tablet_bootstrap.cc:492] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:38.004217 16073 tablet_bootstrap.cc:654] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.004484 16077 tablet_bootstrap.cc:492] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:38.004988 16074 tablet_bootstrap.cc:492] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:38.005582 11644 consensus_queue.cc:237] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:38.005837 16074 tablet_bootstrap.cc:654] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.005995 16075 tablet_bootstrap.cc:492] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:38.006210 16077 tablet_bootstrap.cc:654] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.007488 16074 tablet_bootstrap.cc:492] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:38.007611 16074 ts_tablet_manager.cc:1367] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:38.007627 16075 tablet_bootstrap.cc:654] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.007634 11377 raft_consensus.cc:1261] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:38.008101 16074 raft_consensus.cc:348] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.008270 16076 tablet_bootstrap.cc:492] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:38.008285 11508 raft_consensus.cc:1261] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:38.008335 16074 raft_consensus.cc:374] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.008396 16074 raft_consensus.cc:729] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.008554 16074 consensus_queue.cc:260] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.008622 14191 consensus_queue.cc:1035] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:38.008812 15806 consensus_queue.cc:1035] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:38.009249 16074 ts_tablet_manager.cc:1398] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.009454 16076 tablet_bootstrap.cc:654] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.009698 16074 tablet_bootstrap.cc:492] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:38.010609 16074 tablet_bootstrap.cc:654] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.010752 16075 tablet_bootstrap.cc:492] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:38.010839 16075 ts_tablet_manager.cc:1367] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
W20240501 19:56:38.010877 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d505c415daa54005b8ed22bdc11ce135. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:38.011268 16075 raft_consensus.cc:348] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.011366 16075 raft_consensus.cc:374] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.011394 16075 raft_consensus.cc:729] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.011534 16075 consensus_queue.cc:260] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.012125 16077 tablet_bootstrap.cc:492] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:38.012229 16077 ts_tablet_manager.cc:1367] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:38.012285 15806 raft_consensus.cc:2932] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:38.012610 16077 raft_consensus.cc:348] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.012710 16077 raft_consensus.cc:374] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.012740 16077 raft_consensus.cc:729] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.012749 11377 raft_consensus.cc:2932] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:38.012859 16077 consensus_queue.cc:260] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.015313 16077 ts_tablet_manager.cc:1398] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:38.015552 16074 tablet_bootstrap.cc:492] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:38.015687 16074 ts_tablet_manager.cc:1367] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:38.015537 11104 catalog_manager.cc:5484] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.016218 16075 ts_tablet_manager.cc:1398] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 19:56:38.016201 16074 raft_consensus.cc:348] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.016356 16074 raft_consensus.cc:374] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.016394 16075 tablet_bootstrap.cc:492] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:38.016405 16074 raft_consensus.cc:729] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.016527 16074 consensus_queue.cc:260] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.016811 16074 ts_tablet_manager.cc:1398] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.017270 16075 tablet_bootstrap.cc:654] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.018497 16076 tablet_bootstrap.cc:492] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:38.018627 16076 ts_tablet_manager.cc:1367] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:56:38.019140 16075 tablet_bootstrap.cc:492] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:38.019079 16076 raft_consensus.cc:348] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.019191 16076 raft_consensus.cc:374] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.019269 16076 raft_consensus.cc:729] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.019291 16075 ts_tablet_manager.cc:1367] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:38.019415 16076 consensus_queue.cc:260] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.019810 16076 ts_tablet_manager.cc:1398] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.019902 16075 raft_consensus.cc:348] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.020076 16075 raft_consensus.cc:374] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.020016 11508 raft_consensus.cc:2932] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:38.020152 16075 raft_consensus.cc:729] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.020293 16075 consensus_queue.cc:260] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.020624 16075 ts_tablet_manager.cc:1398] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:38.025377 16076 tablet_bootstrap.cc:492] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:38.026583 16076 tablet_bootstrap.cc:654] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.028141 16073 tablet_bootstrap.cc:492] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:38.028264 16073 ts_tablet_manager.cc:1367] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 19:56:38.029304 16073 raft_consensus.cc:348] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.029435 16073 raft_consensus.cc:374] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.029466 16073 raft_consensus.cc:729] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.029584 16073 consensus_queue.cc:260] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.029961 16073 ts_tablet_manager.cc:1398] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.030182 16073 tablet_bootstrap.cc:492] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:38.032065 16073 tablet_bootstrap.cc:654] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.033138 16073 tablet_bootstrap.cc:492] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:38.033215 16073 ts_tablet_manager.cc:1367] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:38.033593 16073 raft_consensus.cc:348] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.033687 16073 raft_consensus.cc:374] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.033711 16073 raft_consensus.cc:729] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.033823 16073 consensus_queue.cc:260] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.034060 16073 ts_tablet_manager.cc:1398] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:38.039544 16076 tablet_bootstrap.cc:492] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:38.039662 16076 ts_tablet_manager.cc:1367] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 19:56:38.040156 16076 raft_consensus.cc:348] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.040280 16076 raft_consensus.cc:374] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.040314 16076 raft_consensus.cc:729] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.040447 16076 consensus_queue.cc:260] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.040699 16076 ts_tablet_manager.cc:1398] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.046994 11100 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:38.053915 13780 tablet_service.cc:1467] Processing CreateTablet for tablet f65314238a604408a7f0ba3e3e62ced3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7aa498ed1148430e8637a69bc29c296f]), partition=
I20240501 19:56:38.054108 13648 tablet_service.cc:1467] Processing CreateTablet for tablet f65314238a604408a7f0ba3e3e62ced3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7aa498ed1148430e8637a69bc29c296f]), partition=
I20240501 19:56:38.054208 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65314238a604408a7f0ba3e3e62ced3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.054343 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65314238a604408a7f0ba3e3e62ced3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.055718 13912 tablet_service.cc:1467] Processing CreateTablet for tablet f65314238a604408a7f0ba3e3e62ced3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7aa498ed1148430e8637a69bc29c296f]), partition=
I20240501 19:56:38.055954 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65314238a604408a7f0ba3e3e62ced3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.056278 16074 tablet_bootstrap.cc:492] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:38.056695 16110 tablet_bootstrap.cc:492] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:38.057113 16074 tablet_bootstrap.cc:654] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.057461 16110 tablet_bootstrap.cc:654] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.057965 16077 tablet_bootstrap.cc:492] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:38.058055 16074 tablet_bootstrap.cc:492] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:38.058126 16074 ts_tablet_manager.cc:1367] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.058554 16074 raft_consensus.cc:348] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.058653 16074 raft_consensus.cc:374] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.058686 16074 raft_consensus.cc:729] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.058817 16074 consensus_queue.cc:260] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.058976 16077 tablet_bootstrap.cc:654] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.059084 16074 ts_tablet_manager.cc:1398] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.060683 16110 tablet_bootstrap.cc:492] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:38.060773 16110 ts_tablet_manager.cc:1367] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:38.061127 16110 raft_consensus.cc:348] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.061218 16110 raft_consensus.cc:374] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.061249 16110 raft_consensus.cc:729] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.061388 16110 consensus_queue.cc:260] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.061815 16110 ts_tablet_manager.cc:1398] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.062578 16077 tablet_bootstrap.cc:492] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:38.062674 16077 ts_tablet_manager.cc:1367] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 19:56:38.063169 16077 raft_consensus.cc:348] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.063279 16077 raft_consensus.cc:374] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.063313 16077 raft_consensus.cc:729] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.063462 16077 consensus_queue.cc:260] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.063706 16077 ts_tablet_manager.cc:1398] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:38.084625 13437 raft_consensus.cc:482] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:38.084753 13437 raft_consensus.cc:504] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.085219 13437 leader_election.cc:288] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:38.085529 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f03eecaec4b929723c8f02a185606" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:38.085618 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f03eecaec4b929723c8f02a185606" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:38.085677 11377 raft_consensus.cc:2454] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:38.085772 14062 raft_consensus.cc:2454] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:38.086026 11447 leader_election.cc:302] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:38.086261 13437 raft_consensus.cc:2781] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:38.086324 13437 raft_consensus.cc:482] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:38.086364 13437 raft_consensus.cc:3037] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.087255 13437 raft_consensus.cc:504] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.087627 13437 leader_election.cc:288] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:38.087929 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f03eecaec4b929723c8f02a185606" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:38.088084 11377 raft_consensus.cc:3037] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.088130 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f03eecaec4b929723c8f02a185606" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:38.088243 14062 raft_consensus.cc:3037] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.088894 11377 raft_consensus.cc:2454] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:38.088953 14062 raft_consensus.cc:2454] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:38.089216 11445 leader_election.cc:302] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 533908e517b24f77b2af8b5863e4ae5a; no voters:
I20240501 19:56:38.089377 13437 raft_consensus.cc:2781] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.089449 13437 raft_consensus.cc:686] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:38.089573 13437 consensus_queue.cc:237] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.089960 16139 ts_tablet_manager.cc:902] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:38.091106 16139 tablet_copy_client.cc:308] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:38.091634 11400 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet 61cadb3ab334435eaebe99fc3c9e496b from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:59475)
I20240501 19:56:38.091737 11400 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet 61cadb3ab334435eaebe99fc3c9e496b from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:59475: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-61cadb3ab334435eaebe99fc3c9e496b
I20240501 19:56:38.091892 11097 catalog_manager.cc:5484] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.092732 11400 tablet_copy_source_session.cc:205] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:38.093833 16139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cadb3ab334435eaebe99fc3c9e496b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.096004 16139 tablet_copy_client.cc:783] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:38.096227 16139 tablet_copy_client.cc:647] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:38.099476 16139 tablet_copy_client.cc:523] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:38.100735 16139 tablet_bootstrap.cc:492] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:38.104974 16156 ts_tablet_manager.cc:902] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:38.106580 16156 tablet_copy_client.cc:308] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:38.107132 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet 1772073172764f8db95c597f8da14f12 from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:33205)
I20240501 19:56:38.107244 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet 1772073172764f8db95c597f8da14f12 from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:33205: session id = 6921cf2e41f64d66842885512da8de6c-1772073172764f8db95c597f8da14f12
I20240501 19:56:38.108497 11664 tablet_copy_source_session.cc:205] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:38.109439 16156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1772073172764f8db95c597f8da14f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.111223 16156 tablet_copy_client.cc:783] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:38.111346 16156 tablet_copy_client.cc:647] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:38.113008 16156 tablet_copy_client.cc:523] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:38.114391 16156 tablet_bootstrap.cc:492] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:38.127997 16139 tablet_bootstrap.cc:492] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:38.128458 16139 tablet_bootstrap.cc:492] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:38.128646 16139 ts_tablet_manager.cc:1367] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20240501 19:56:38.129225 16139 raft_consensus.cc:348] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:38.129460 16139 raft_consensus.cc:729] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:38.129635 16139 consensus_queue.cc:260] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:38.130513 16139 ts_tablet_manager.cc:1398] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.130914 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-61cadb3ab334435eaebe99fc3c9e496b received from {username='slave'} at 127.5.40.6:59475
I20240501 19:56:38.131001 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-61cadb3ab334435eaebe99fc3c9e496b on tablet 61cadb3ab334435eaebe99fc3c9e496b with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:38.137102 15806 raft_consensus.cc:1050] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:38.137560 15806 consensus_queue.cc:237] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:38.138510 16156 tablet_bootstrap.cc:492] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:38.138866 16156 tablet_bootstrap.cc:492] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:38.139088 16156 ts_tablet_manager.cc:1367] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.011s
I20240501 19:56:38.139570 16156 raft_consensus.cc:348] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:38.139780 16156 raft_consensus.cc:729] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:38.139789 13667 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:38.139963 16156 consensus_queue.cc:260] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:38.140358 16156 ts_tablet_manager.cc:1398] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.140357 11508 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:38.140892 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-1772073172764f8db95c597f8da14f12 received from {username='slave'} at 127.5.40.7:33205
I20240501 19:56:38.140966 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-1772073172764f8db95c597f8da14f12 on tablet 1772073172764f8db95c597f8da14f12 with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:38.141533 14062 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:38.142024 15081 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:38.142357 15081 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:38.142640 15081 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:38.144966 15806 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:38.146040 14062 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:38.147409 11508 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:38.147667 13667 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:38.148160 11100 catalog_manager.cc:5484] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:38.151798 15815 raft_consensus.cc:1050] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:38.152264 15815 consensus_queue.cc:237] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:38.153645 11508 raft_consensus.cc:1261] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.153707 13799 raft_consensus.cc:1261] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.154052 16188 consensus_queue.cc:1035] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.154227 15815 consensus_queue.cc:1035] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.154690 11643 raft_consensus.cc:1261] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.156646 15815 consensus_queue.cc:1035] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.161129 16198 ts_tablet_manager.cc:902] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:38.161703 16198 tablet_copy_client.cc:308] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:38.162175 15815 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.162549 11643 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.163599 11644 consensus_queue.cc:237] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:38.165035 13800 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.165946 13667 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:38.166040 11104 catalog_manager.cc:5484] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:38.166581 11508 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.167644 15081 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.166838 14062 raft_consensus.cc:1261] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:38.170425 15081 consensus_queue.cc:1035] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.172901 15081 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:38.174813 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e307d45fb57f444ca09d94d59b9bcd64 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:38.174958 14062 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:38.177160 11247 consensus_queue.cc:237] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:38.177881 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 5e63472cfff449089d88d18b9e495912 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:54331)
I20240501 19:56:38.177981 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 5e63472cfff449089d88d18b9e495912 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:54331: session id = 533908e517b24f77b2af8b5863e4ae5a-5e63472cfff449089d88d18b9e495912
I20240501 19:56:38.178930 11533 tablet_copy_source_session.cc:205] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:38.179078 13800 raft_consensus.cc:1261] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:38.179286 13667 raft_consensus.cc:2932] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:38.179494 11643 raft_consensus.cc:1261] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:38.179306 15930 raft_consensus.cc:482] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:38.179881 16188 consensus_queue.cc:1035] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:38.179917 15930 raft_consensus.cc:504] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.180212 16188 consensus_queue.cc:1035] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:38.180255 15930 leader_election.cc:288] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:38.180734 16198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e63472cfff449089d88d18b9e495912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.180881 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b69cb1df2543f583eec2529a292897" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:38.181025 11377 raft_consensus.cc:2454] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:38.181325 13605 leader_election.cc:302] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:38.182118 15930 raft_consensus.cc:2781] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:38.182171 15930 raft_consensus.cc:482] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:38.182149 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b69cb1df2543f583eec2529a292897" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:38.182207 15930 raft_consensus.cc:3037] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.182272 14062 raft_consensus.cc:2454] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:38.183020 16198 tablet_copy_client.cc:783] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:38.183089 15930 raft_consensus.cc:504] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.183156 16198 tablet_copy_client.cc:647] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:38.183482 15930 leader_election.cc:288] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:38.184144 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b69cb1df2543f583eec2529a292897" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:38.184257 11377 raft_consensus.cc:3037] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.186262 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b69cb1df2543f583eec2529a292897" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:38.186864 11104 catalog_manager.cc:5484] T e307d45fb57f444ca09d94d59b9bcd64 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:38.187171 13434 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.187012 11377 raft_consensus.cc:2454] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:38.187716 13605 leader_election.cc:302] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:38.186383 14062 raft_consensus.cc:3037] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.187140 16198 tablet_copy_client.cc:523] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:38.189110 16198 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:38.189318 15930 raft_consensus.cc:2781] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.189424 15930 raft_consensus.cc:686] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:38.189556 15930 consensus_queue.cc:237] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.188671 14062 raft_consensus.cc:2454] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:38.190529 11643 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.192198 13800 raft_consensus.cc:2932] T e96e8e7c66584b1198bea464e5a1a4d8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.192624 11097 catalog_manager.cc:5484] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.193259 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e96e8e7c66584b1198bea464e5a1a4d8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:38.194646 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet e307d45fb57f444ca09d94d59b9bcd64 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:38.195354 14711 tablet_replica.cc:330] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:38.195526 14711 raft_consensus.cc:2229] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.195727 14711 raft_consensus.cc:2258] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.196347 14711 ts_tablet_manager.cc:1875] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:38.197368 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet e96e8e7c66584b1198bea464e5a1a4d8 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:38.198362 11105 catalog_manager.cc:5484] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.198596 14711 ts_tablet_manager.cc:1888] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:38.198657 14711 log.cc:1196] T e307d45fb57f444ca09d94d59b9bcd64 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/e307d45fb57f444ca09d94d59b9bcd64
I20240501 19:56:38.199054 14711 tablet_replica.cc:330] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:38.199234 14711 raft_consensus.cc:2229] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.199424 14711 raft_consensus.cc:2258] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.199569 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet e307d45fb57f444ca09d94d59b9bcd64 (table rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]) successfully deleted
I20240501 19:56:38.200188 14711 ts_tablet_manager.cc:1875] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:38.202152 14711 ts_tablet_manager.cc:1888] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:38.202406 14711 log.cc:1196] T e96e8e7c66584b1198bea464e5a1a4d8 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/e96e8e7c66584b1198bea464e5a1a4d8
I20240501 19:56:38.202522 14934 raft_consensus.cc:1050] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:38.203048 14934 consensus_queue.cc:237] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:38.203554 16209 ts_tablet_manager.cc:902] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: Initiating tablet copy from peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:38.204337 16209 tablet_copy_client.cc:308] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.6:46331
I20240501 19:56:38.204717 11508 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.204728 11643 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.204954 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet e96e8e7c66584b1198bea464e5a1a4d8 (table rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]) successfully deleted
I20240501 19:56:38.205229 13821 tablet_copy_service.cc:140] P 96fc18dbb5fd41afb9bebf805e8d33a7: Received BeginTabletCopySession request for tablet 87aabbb9dc2440f28a3b6944dea38fe7 from peer ef60ab93b49c4346af4f62d3b98558b8 ({username='slave'} at 127.5.40.2:37395)
I20240501 19:56:38.205235 15721 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.205327 13821 tablet_copy_service.cc:161] P 96fc18dbb5fd41afb9bebf805e8d33a7: Beginning new tablet copy session on tablet 87aabbb9dc2440f28a3b6944dea38fe7 from peer ef60ab93b49c4346af4f62d3b98558b8 at {username='slave'} at 127.5.40.2:37395: session id = ef60ab93b49c4346af4f62d3b98558b8-87aabbb9dc2440f28a3b6944dea38fe7
I20240501 19:56:38.205691 14473 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.206203 13667 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.206275 13821 tablet_copy_source_session.cc:205] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:38.208074 14473 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.211727 15094 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:38.212131 16209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aabbb9dc2440f28a3b6944dea38fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.213702 13667 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:38.214429 16209 tablet_copy_client.cc:783] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:38.214596 16209 tablet_copy_client.cc:647] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:38.216480 15462 raft_consensus.cc:1050] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:38.216892 15462 consensus_queue.cc:237] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:38.217536 11643 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:38.220335 11508 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:38.222607 13931 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.223664 13434 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.223258 11105 catalog_manager.cc:5484] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:38.226792 16209 tablet_copy_client.cc:523] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:38.227519 11508 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.227936 11377 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.227921 13434 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.228694 16209 tablet_bootstrap.cc:492] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:38.229866 13434 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.236923 16209 tablet_bootstrap.cc:492] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:38.237222 16209 tablet_bootstrap.cc:492] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap complete.
I20240501 19:56:38.237500 16209 ts_tablet_manager.cc:1367] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240501 19:56:38.238056 16209 raft_consensus.cc:348] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:38.238277 16209 raft_consensus.cc:729] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: LEARNER
I20240501 19:56:38.238394 16209 consensus_queue.cc:260] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:38.238765 16209 ts_tablet_manager.cc:1398] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:38.239384 13821 tablet_copy_service.cc:342] P 96fc18dbb5fd41afb9bebf805e8d33a7: Request end of tablet copy session ef60ab93b49c4346af4f62d3b98558b8-87aabbb9dc2440f28a3b6944dea38fe7 received from {username='slave'} at 127.5.40.2:37395
I20240501 19:56:38.239488 13821 tablet_copy_service.cc:433] P 96fc18dbb5fd41afb9bebf805e8d33a7: ending tablet copy session ef60ab93b49c4346af4f62d3b98558b8-87aabbb9dc2440f28a3b6944dea38fe7 on tablet 87aabbb9dc2440f28a3b6944dea38fe7 with peer ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:38.241416 16198 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:38.241760 16198 tablet_bootstrap.cc:492] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:38.241956 16198 ts_tablet_manager.cc:1367] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.053s user 0.026s sys 0.000s
I20240501 19:56:38.242435 16198 raft_consensus.cc:348] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:38.242475 13434 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.243131 16198 raft_consensus.cc:729] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:38.243305 16198 consensus_queue.cc:260] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:38.244135 16198 ts_tablet_manager.cc:1398] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:38.244685 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-5e63472cfff449089d88d18b9e495912 received from {username='slave'} at 127.5.40.8:54331
I20240501 19:56:38.244764 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-5e63472cfff449089d88d18b9e495912 on tablet 5e63472cfff449089d88d18b9e495912 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:38.245223 14935 raft_consensus.cc:482] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:38.245340 14935 raft_consensus.cc:504] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.245584 13931 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.245749 14935 leader_election.cc:288] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:38.246243 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65314238a604408a7f0ba3e3e62ced3" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:38.246205 11508 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.246380 13932 raft_consensus.cc:2454] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:38.247382 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65314238a604408a7f0ba3e3e62ced3" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:38.247540 13667 raft_consensus.cc:2454] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:38.247671 13734 leader_election.cc:302] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:38.247552 11377 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.247993 15507 raft_consensus.cc:2781] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:38.248044 15507 raft_consensus.cc:482] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:38.248076 15507 raft_consensus.cc:3037] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.248874 15507 raft_consensus.cc:504] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.249213 15507 leader_election.cc:288] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:38.250420 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65314238a604408a7f0ba3e3e62ced3" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:38.250432 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65314238a604408a7f0ba3e3e62ced3" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:38.250536 13932 raft_consensus.cc:3037] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.250536 13667 raft_consensus.cc:3037] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.250527 11097 catalog_manager.cc:5484] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.251559 13667 raft_consensus.cc:2454] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:38.252261 13734 leader_election.cc:302] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:38.252547 14935 raft_consensus.cc:2781] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.252619 14935 raft_consensus.cc:686] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:38.252737 14935 consensus_queue.cc:237] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.254784 11097 catalog_manager.cc:5484] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.255096 13932 raft_consensus.cc:2454] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:38.270702 15930 raft_consensus.cc:482] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:38.270829 15930 raft_consensus.cc:504] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.271272 15930 leader_election.cc:288] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:38.271764 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811d018fde314227a378f2d530e0a924" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:38.271905 11508 raft_consensus.cc:2454] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:38.271950 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811d018fde314227a378f2d530e0a924" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:38.272102 13932 raft_consensus.cc:2454] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:38.272378 13603 leader_election.cc:302] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:38.272758 15930 raft_consensus.cc:2781] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:38.272827 15930 raft_consensus.cc:482] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:38.272864 15930 raft_consensus.cc:3037] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.273993 15930 raft_consensus.cc:504] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.274396 15930 leader_election.cc:288] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:38.276011 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811d018fde314227a378f2d530e0a924" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:38.276016 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811d018fde314227a378f2d530e0a924" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:38.276144 13932 raft_consensus.cc:3037] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.276149 11508 raft_consensus.cc:3037] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.277019 13932 raft_consensus.cc:2454] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:38.277087 11508 raft_consensus.cc:2454] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:38.277374 13603 leader_election.cc:302] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:38.277546 15930 raft_consensus.cc:2781] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.277637 15930 raft_consensus.cc:686] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:38.277799 15930 consensus_queue.cc:237] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:38.279973 11097 catalog_manager.cc:5484] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.315377 11104 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 19:56:38.315585 11104 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 19:56:38.318233 11104 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 01ed8143e5454a4aaef3e142069f35a2
I20240501 19:56:38.318586 11104 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 4 replicas of tablet b7073666bc3f4c0a8c1e003fb1ef4fef
I20240501 19:56:38.318874 11104 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 4 replicas of tablet c31cf707a7e84f139fce27df22b376fd
I20240501 19:56:38.319301 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ed8143e5454a4aaef3e142069f35a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:38.319316 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ed8143e5454a4aaef3e142069f35a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:38.319308 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet b7073666bc3f4c0a8c1e003fb1ef4fef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:38.319559 15731 tablet_replica.cc:330] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:38.319697 15731 raft_consensus.cc:2229] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.319705 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet c31cf707a7e84f139fce27df22b376fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:38.319870 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet b7073666bc3f4c0a8c1e003fb1ef4fef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:38.319892 15731 raft_consensus.cc:2258] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.319957 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet c31cf707a7e84f139fce27df22b376fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:38.320091 15964 tablet_replica.cc:330] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:38.320163 15732 tablet_replica.cc:330] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:38.320211 15964 raft_consensus.cc:2229] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.320297 15732 raft_consensus.cc:2229] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.320356 15964 raft_consensus.cc:2258] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.320326 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet c31cf707a7e84f139fce27df22b376fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:38.320418 15732 raft_consensus.cc:2258] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.320515 14261 tablet_replica.cc:330] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:38.320590 15095 tablet_replica.cc:330] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:38.320588 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet b7073666bc3f4c0a8c1e003fb1ef4fef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:38.320637 14261 raft_consensus.cc:2229] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.320711 15095 raft_consensus.cc:2229] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.320791 14261 raft_consensus.cc:2258] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.320889 15095 raft_consensus.cc:2258] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.320902 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ed8143e5454a4aaef3e142069f35a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:38.321084 13648 tablet_service.cc:1514] Processing DeleteTablet for tablet b7073666bc3f4c0a8c1e003fb1ef4fef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:38.321348 14711 tablet_replica.cc:330] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:38.321337 14261 ts_tablet_manager.cc:1875] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.321507 14711 raft_consensus.cc:2229] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:38.321593 15095 ts_tablet_manager.cc:1875] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.321759 14711 raft_consensus.cc:2258] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.322542 14711 ts_tablet_manager.cc:1875] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.323781 14261 ts_tablet_manager.cc:1888] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:38.323858 14261 log.cc:1196] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/c31cf707a7e84f139fce27df22b376fd
I20240501 19:56:38.323868 15095 ts_tablet_manager.cc:1888] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.7
I20240501 19:56:38.323882 15964 ts_tablet_manager.cc:1875] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.323935 15095 log.cc:1196] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/b7073666bc3f4c0a8c1e003fb1ef4fef
I20240501 19:56:38.324101 14261 ts_tablet_manager.cc:1909] T c31cf707a7e84f139fce27df22b376fd P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:38.324170 15095 ts_tablet_manager.cc:1909] T b7073666bc3f4c0a8c1e003fb1ef4fef P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:38.324682 14261 tablet_replica.cc:330] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:38.324683 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet c31cf707a7e84f139fce27df22b376fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:38.324798 14261 raft_consensus.cc:2229] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:38.324995 14261 raft_consensus.cc:2258] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.324997 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet c31cf707a7e84f139fce27df22b376fd (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.325227 14711 ts_tablet_manager.cc:1888] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.7
I20240501 19:56:38.325289 14711 log.cc:1196] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/b7073666bc3f4c0a8c1e003fb1ef4fef
W20240501 19:56:38.325634 11089 catalog_manager.cc:4797] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): delete failed for tablet b7073666bc3f4c0a8c1e003fb1ef4fef because the tablet was not found. No further retry: Not found: Tablet not found: b7073666bc3f4c0a8c1e003fb1ef4fef
I20240501 19:56:38.326050 16285 ts_tablet_manager.cc:902] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: Initiating tablet copy from peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:38.326102 15964 ts_tablet_manager.cc:1888] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.7
I20240501 19:56:38.326162 15964 log.cc:1196] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/b7073666bc3f4c0a8c1e003fb1ef4fef
I20240501 19:56:38.326397 15964 ts_tablet_manager.cc:1909] T b7073666bc3f4c0a8c1e003fb1ef4fef P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:38.326491 15732 ts_tablet_manager.cc:1875] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.326879 15930 raft_consensus.cc:1050] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: attempting to promote NON_VOTER 08711534813d47ba85aaec0771ed5d0f to VOTER
I20240501 19:56:38.326872 16285 tablet_copy_client.cc:308] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.5:42501
I20240501 19:56:38.327414 15930 consensus_queue.cc:237] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:38.327636 16139 ts_tablet_manager.cc:902] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:38.328081 16139 tablet_copy_client.cc:308] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:38.328302 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet b7073666bc3f4c0a8c1e003fb1ef4fef (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.327237 15731 ts_tablet_manager.cc:1875] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.328644 15732 ts_tablet_manager.cc:1888] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:38.328706 15732 log.cc:1196] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/01ed8143e5454a4aaef3e142069f35a2
I20240501 19:56:38.328922 15732 ts_tablet_manager.cc:1909] T 01ed8143e5454a4aaef3e142069f35a2 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:38.329175 13800 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:38.329295 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet b7073666bc3f4c0a8c1e003fb1ef4fef (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.329365 11400 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet cebd15701ee44b11a2f8804740fb4299 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:59475)
I20240501 19:56:38.329459 11400 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet cebd15701ee44b11a2f8804740fb4299 from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:59475: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-cebd15701ee44b11a2f8804740fb4299
I20240501 19:56:38.329641 15931 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:38.330194 11508 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:38.330483 11400 tablet_copy_source_session.cc:205] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:38.330615 15731 ts_tablet_manager.cc:1888] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:38.330677 15731 log.cc:1196] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/01ed8143e5454a4aaef3e142069f35a2
I20240501 19:56:38.330716 14711 ts_tablet_manager.cc:1909] T b7073666bc3f4c0a8c1e003fb1ef4fef P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:38.330955 13932 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:38.330976 15931 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:38.331080 13690 tablet_copy_service.cc:140] P 8125d8ad6d0d44caae9c5c90c90df96b: Received BeginTabletCopySession request for tablet 96511455168148b7b619670f635d3a3e from peer 8ad18cd54c6340fd92b874157875fa49 ({username='slave'} at 127.5.40.4:35171)
I20240501 19:56:38.331127 15731 ts_tablet_manager.cc:1909] T 01ed8143e5454a4aaef3e142069f35a2 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:38.331156 13690 tablet_copy_service.cc:161] P 8125d8ad6d0d44caae9c5c90c90df96b: Beginning new tablet copy session on tablet 96511455168148b7b619670f635d3a3e from peer 8ad18cd54c6340fd92b874157875fa49 at {username='slave'} at 127.5.40.4:35171: session id = 8ad18cd54c6340fd92b874157875fa49-96511455168148b7b619670f635d3a3e
I20240501 19:56:38.331801 16139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cebd15701ee44b11a2f8804740fb4299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.331851 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 01ed8143e5454a4aaef3e142069f35a2 (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.332087 15731 tablet_replica.cc:330] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:38.332211 15731 raft_consensus.cc:2229] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.332242 13690 tablet_copy_source_session.cc:205] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:38.332350 15731 raft_consensus.cc:2258] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.332939 15095 tablet_replica.cc:330] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:38.333073 15095 raft_consensus.cc:2229] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:38.333379 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet b7073666bc3f4c0a8c1e003fb1ef4fef (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.333489 16285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96511455168148b7b619670f635d3a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.333817 15731 ts_tablet_manager.cc:1875] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.333886 16139 tablet_copy_client.cc:783] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:38.334012 16139 tablet_copy_client.cc:647] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:38.333969 16296 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:38.335261 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet 01ed8143e5454a4aaef3e142069f35a2 (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.335736 15731 ts_tablet_manager.cc:1888] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:38.333318 15095 raft_consensus.cc:2258] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.336442 15095 ts_tablet_manager.cc:1875] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.336473 14261 ts_tablet_manager.cc:1875] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.337559 16296 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:38.338506 14261 ts_tablet_manager.cc:1888] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 19:56:38.338506 15095 ts_tablet_manager.cc:1888] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:38.338583 15095 log.cc:1196] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/c31cf707a7e84f139fce27df22b376fd
I20240501 19:56:38.335796 16285 tablet_copy_client.cc:783] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:38.338778 16139 tablet_copy_client.cc:523] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:38.338783 16285 tablet_copy_client.cc:647] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:38.338945 15731 log.cc:1196] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/c31cf707a7e84f139fce27df22b376fd
I20240501 19:56:38.339260 15731 ts_tablet_manager.cc:1909] T c31cf707a7e84f139fce27df22b376fd P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:38.339183 13800 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:38.340128 16139 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:38.340128 15095 ts_tablet_manager.cc:1909] T c31cf707a7e84f139fce27df22b376fd P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting consensus metadata
I20240501 19:56:38.340885 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet c31cf707a7e84f139fce27df22b376fd (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.341027 11104 catalog_manager.cc:5484] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:38.341259 16283 tablet_replica.cc:330] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:38.341401 16283 raft_consensus.cc:2229] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Raft consensus shutting down.
I20240501 19:56:38.341536 16283 raft_consensus.cc:2258] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Raft consensus is shut down!
I20240501 19:56:38.342123 16283 ts_tablet_manager.cc:1875] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.342357 13932 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:38.343516 14261 log.cc:1196] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/01ed8143e5454a4aaef3e142069f35a2
I20240501 19:56:38.343796 11508 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:38.344504 16283 ts_tablet_manager.cc:1888] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:38.344560 14261 ts_tablet_manager.cc:1909] T 01ed8143e5454a4aaef3e142069f35a2 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:38.344568 16283 log.cc:1196] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/c31cf707a7e84f139fce27df22b376fd
I20240501 19:56:38.344782 16283 ts_tablet_manager.cc:1909] T c31cf707a7e84f139fce27df22b376fd P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:38.346431 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet c31cf707a7e84f139fce27df22b376fd (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.347194 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 01ed8143e5454a4aaef3e142069f35a2 (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.348085 13437 raft_consensus.cc:1050] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:38.348320 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet c31cf707a7e84f139fce27df22b376fd (table rebalance_test_table_18 [id=207558d33b6d45509cde97b403b849da]) successfully deleted
I20240501 19:56:38.348493 13437 consensus_queue.cc:237] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:38.350474 11377 raft_consensus.cc:1261] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.350729 16285 tablet_copy_client.cc:523] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:38.351066 11247 raft_consensus.cc:1261] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.351164 13155 consensus_queue.cc:1035] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.351938 13155 consensus_queue.cc:1035] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.351970 13800 raft_consensus.cc:1261] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.352715 16285 tablet_bootstrap.cc:492] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:38.353503 13155 consensus_queue.cc:1035] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.354496 13155 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.356627 11377 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.360774 11247 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.361634 16285 tablet_bootstrap.cc:492] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:38.361963 16285 tablet_bootstrap.cc:492] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: Bootstrap complete.
I20240501 19:56:38.361927 11100 catalog_manager.cc:5484] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.362123 16285 ts_tablet_manager.cc:1367] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 19:56:38.362620 16285 raft_consensus.cc:348] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:38.362939 16285 raft_consensus.cc:729] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: LEARNER
I20240501 19:56:38.363411 16285 consensus_queue.cc:260] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:38.363895 16285 ts_tablet_manager.cc:1398] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.365386 13690 tablet_copy_service.cc:342] P 8125d8ad6d0d44caae9c5c90c90df96b: Request end of tablet copy session 8ad18cd54c6340fd92b874157875fa49-96511455168148b7b619670f635d3a3e received from {username='slave'} at 127.5.40.4:35171
I20240501 19:56:38.365487 13690 tablet_copy_service.cc:433] P 8125d8ad6d0d44caae9c5c90c90df96b: ending tablet copy session 8ad18cd54c6340fd92b874157875fa49-96511455168148b7b619670f635d3a3e on tablet 96511455168148b7b619670f635d3a3e with peer 8ad18cd54c6340fd92b874157875fa49
I20240501 19:56:38.367189 13800 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.375239 11508 consensus_queue.cc:237] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:38.376821 11247 raft_consensus.cc:1261] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:38.376889 16139 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:38.377169 16139 tablet_bootstrap.cc:492] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:38.377414 16139 ts_tablet_manager.cc:1367] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.037s user 0.023s sys 0.004s
I20240501 19:56:38.377434 15765 consensus_queue.cc:1035] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:38.378015 16139 raft_consensus.cc:348] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:38.378224 16139 raft_consensus.cc:729] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:38.378379 16139 consensus_queue.cc:260] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:38.378795 13800 raft_consensus.cc:1261] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:38.379258 15765 consensus_queue.cc:1035] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:38.379323 16139 ts_tablet_manager.cc:1398] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.379776 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-cebd15701ee44b11a2f8804740fb4299 received from {username='slave'} at 127.5.40.6:59475
I20240501 19:56:38.379858 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-cebd15701ee44b11a2f8804740fb4299 on tablet cebd15701ee44b11a2f8804740fb4299 with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:38.381299 16302 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.381654 11247 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.383841 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c86f45702e94c0e898e4e81148f2098 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:38.385012 11097 catalog_manager.cc:5484] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.388552 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c86f45702e94c0e898e4e81148f2098 with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:38.388669 13800 raft_consensus.cc:2932] T 2c86f45702e94c0e898e4e81148f2098 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.388854 14261 tablet_replica.cc:330] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:38.389029 14261 raft_consensus.cc:2229] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.389217 14261 raft_consensus.cc:2258] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.389902 14261 ts_tablet_manager.cc:1875] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:38.392238 14261 ts_tablet_manager.cc:1888] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:38.392310 14261 log.cc:1196] T 2c86f45702e94c0e898e4e81148f2098 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/2c86f45702e94c0e898e4e81148f2098
I20240501 19:56:38.392932 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 2c86f45702e94c0e898e4e81148f2098 (table rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]) successfully deleted
I20240501 19:56:38.395789 11247 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49"
dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47142
I20240501 19:56:38.395977 11247 raft_consensus.cc:595] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.399968 11097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:38.403962 13932 raft_consensus.cc:1261] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.404266 13667 raft_consensus.cc:1261] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.404659 15507 consensus_queue.cc:1035] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.404829 14935 consensus_queue.cc:1035] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.407158 11097 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet f65314238a604408a7f0ba3e3e62ced3 (table rebalancer_dynamic_table [id=7aa498ed1148430e8637a69bc29c296f]). Expected version 1 got 0
I20240501 19:56:38.408021 11106 catalog_manager.cc:5576] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501) does not have the latest schema for tablet f65314238a604408a7f0ba3e3e62ced3 (table rebalancer_dynamic_table [id=7aa498ed1148430e8637a69bc29c296f]). Expected version 1 got 0
I20240501 19:56:38.410658 16333 tablet.cc:1722] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:38.411008 16333 tablet.cc:1620] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:38.413780 11104 catalog_manager.cc:5576] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501) does not have the latest schema for tablet f65314238a604408a7f0ba3e3e62ced3 (table rebalancer_dynamic_table [id=7aa498ed1148430e8637a69bc29c296f]). Expected version 1 got 0
I20240501 19:56:38.414496 11097 catalog_manager.cc:5576] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331) does not have the latest schema for tablet f65314238a604408a7f0ba3e3e62ced3 (table rebalancer_dynamic_table [id=7aa498ed1148430e8637a69bc29c296f]). Expected version 1 got 0
I20240501 19:56:38.414758 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=7aa498ed1148430e8637a69bc29c296f] alter complete (version 1)
I20240501 19:56:38.415688 16332 tablet.cc:1722] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:38.416198 16334 tablet.cc:1722] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:38.416612 16334 tablet.cc:1620] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:38.416009 16332 tablet.cc:1620] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:38.425660 15765 raft_consensus.cc:1050] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:38.426113 15765 consensus_queue.cc:237] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:38.427392 11380 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.427686 11247 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.427733 13800 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.428165 15765 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.428349 16302 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.428496 13437 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.431074 16302 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.431520 13800 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.431931 11247 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.434127 11097 catalog_manager.cc:5484] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:38.434185 11380 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.437606 11508 consensus_queue.cc:237] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:38.438860 11247 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:38.438973 11380 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb"
dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51800
I20240501 19:56:38.439043 13800 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:38.439119 11380 raft_consensus.cc:595] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.439217 13437 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:38.440166 13437 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:38.440447 14794 raft_consensus.cc:1050] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7: attempting to promote NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc to VOTER
I20240501 19:56:38.440882 14794 consensus_queue.cc:237] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } }
I20240501 19:56:38.442256 13667 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:38.442492 11644 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:38.442800 11245 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:38.443171 15507 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:38.443521 15507 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:38.443481 13155 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.444022 14794 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:38.444240 13800 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.445042 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3b35dab7eec4b73b208f687dad46ba2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:38.445253 11247 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.447003 11104 catalog_manager.cc:5484] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:38.449182 14794 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:38.449821 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b35dab7eec4b73b208f687dad46ba2 with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:38.450189 14261 tablet_replica.cc:330] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:38.450346 14261 raft_consensus.cc:2229] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.450531 14261 raft_consensus.cc:2258] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.451241 14261 ts_tablet_manager.cc:1875] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:38.451272 11245 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:38.451893 11644 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:38.453001 13667 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:38.453506 14261 ts_tablet_manager.cc:1888] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:38.453569 14261 log.cc:1196] T c3b35dab7eec4b73b208f687dad46ba2 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3b35dab7eec4b73b208f687dad46ba2
I20240501 19:56:38.454121 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet c3b35dab7eec4b73b208f687dad46ba2 (table rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]) successfully deleted
I20240501 19:56:38.455719 11104 catalog_manager.cc:5484] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:38.459720 13800 consensus_queue.cc:237] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } }
I20240501 19:56:38.460840 11245 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:38.460876 13669 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:38.461256 16352 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:38.461450 14935 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:38.463508 14794 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:38.463879 13669 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:38.464253 11245 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:38.465085 11092 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8078e0cbffa943c98584224d05b0ece0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:38.466418 11106 catalog_manager.cc:5484] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 3 to 4, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:38.468924 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 8078e0cbffa943c98584224d05b0ece0 with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:38.469161 15731 tablet_replica.cc:330] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:38.469307 15731 raft_consensus.cc:2229] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.469439 15731 raft_consensus.cc:2258] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.470177 15731 ts_tablet_manager.cc:1875] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:38.472402 15731 ts_tablet_manager.cc:1888] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:38.472468 15731 log.cc:1196] T 8078e0cbffa943c98584224d05b0ece0 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/8078e0cbffa943c98584224d05b0ece0
I20240501 19:56:38.473016 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 8078e0cbffa943c98584224d05b0ece0 (table rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]) successfully deleted
I20240501 19:56:38.473470 13669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be"
dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35364
I20240501 19:56:38.473624 13669 raft_consensus.cc:595] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.481254 11644 consensus_queue.cc:237] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:38.482684 13932 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:38.482684 11508 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:38.482707 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69facf86bba24f209c06afafc1ba8f7b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:38.483191 16178 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:38.483382 15806 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:38.485260 14191 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:38.485692 11508 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:38.485730 13932 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:38.488492 11100 catalog_manager.cc:5484] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:38.509227 16302 consensus_queue.cc:1035] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.512708 16302 consensus_queue.cc:1035] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.519151 11106 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:38.523090 16352 raft_consensus.cc:1050] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:38.523552 16352 consensus_queue.cc:237] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:38.525090 13669 raft_consensus.cc:1261] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:38.525260 14062 raft_consensus.cc:1261] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:38.525629 13932 raft_consensus.cc:1261] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:38.525643 16332 tablet.cc:1722] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:38.525947 16332 tablet.cc:1620] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:38.526600 14935 consensus_queue.cc:1035] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:38.526763 14794 consensus_queue.cc:1035] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:38.526829 16333 tablet.cc:1722] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:38.526867 15507 consensus_queue.cc:1035] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:38.527105 16333 tablet.cc:1620] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:38.529625 16334 tablet.cc:1722] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:38.529942 16334 tablet.cc:1620] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:38.531107 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=7aa498ed1148430e8637a69bc29c296f] alter complete (version 2)
I20240501 19:56:38.533895 14794 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.535072 14062 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.537420 13667 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.537597 11106 catalog_manager.cc:5484] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 2 to 3, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.537997 13932 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.542531 13800 consensus_queue.cc:237] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:38.543792 13932 raft_consensus.cc:1261] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:38.543916 14062 raft_consensus.cc:1261] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:38.544271 14935 consensus_queue.cc:1035] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:38.544464 16352 consensus_queue.cc:1035] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:38.546408 16352 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.546989 13932 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.547044 14062 raft_consensus.cc:2932] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:38.548285 11092 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fc692e320d94825964040105a3175cd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:38.549031 11097 catalog_manager.cc:5484] T 1fc692e320d94825964040105a3175cd P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 3 to 4, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:38.554678 13648 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc692e320d94825964040105a3175cd with delete_type TABLET_DATA_TOMBSTONED (TS 8125d8ad6d0d44caae9c5c90c90df96b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:38.554997 16284 tablet_replica.cc:330] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:38.555142 16284 raft_consensus.cc:2229] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.555291 16284 raft_consensus.cc:2258] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.555953 16284 ts_tablet_manager.cc:1875] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:38.558342 16284 ts_tablet_manager.cc:1888] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:38.558425 16284 log.cc:1196] T 1fc692e320d94825964040105a3175cd P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/1fc692e320d94825964040105a3175cd
I20240501 19:56:38.559213 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet 1fc692e320d94825964040105a3175cd (table rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]) successfully deleted
I20240501 19:56:38.595485 13800 raft_consensus.cc:1203] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:38.597791 11245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49"
dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47164
I20240501 19:56:38.597945 11245 raft_consensus.cc:595] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.604455 16296 consensus_queue.cc:1035] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.605799 16156 ts_tablet_manager.cc:902] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:38.606315 16156 tablet_copy_client.cc:308] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:38.606803 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet d505c415daa54005b8ed22bdc11ce135 from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:33205)
I20240501 19:56:38.606909 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet d505c415daa54005b8ed22bdc11ce135 from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:33205: session id = 6921cf2e41f64d66842885512da8de6c-d505c415daa54005b8ed22bdc11ce135
I20240501 19:56:38.607950 11664 tablet_copy_source_session.cc:205] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:38.609555 15930 consensus_queue.cc:1035] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.610714 16156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d505c415daa54005b8ed22bdc11ce135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.613116 16156 tablet_copy_client.cc:783] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:38.613260 16156 tablet_copy_client.cc:647] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:38.614387 11377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb"
dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51840
I20240501 19:56:38.614506 11377 raft_consensus.cc:595] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.614823 16220 raft_consensus.cc:979] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: : Instructing follower 08711534813d47ba85aaec0771ed5d0f to start an election
I20240501 19:56:38.614909 16220 raft_consensus.cc:1067] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Signalling peer 08711534813d47ba85aaec0771ed5d0f to start an election
I20240501 19:56:38.615159 16156 tablet_copy_client.cc:523] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:38.615651 11508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb"
dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
from {username='slave'} at 127.5.40.2:37613
I20240501 19:56:38.615801 11508 raft_consensus.cc:482] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:38.615870 11508 raft_consensus.cc:3037] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.616770 16156 tablet_bootstrap.cc:492] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:38.617902 11508 raft_consensus.cc:504] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:38.618441 11508 leader_election.cc:288] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:38.619199 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:38.619318 11643 raft_consensus.cc:3037] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.620188 11643 raft_consensus.cc:2454] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 2.
I20240501 19:56:38.620456 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:38.620571 11377 raft_consensus.cc:3032] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:38.620609 11377 raft_consensus.cc:729] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:38.620788 11377 consensus_queue.cc:260] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:38.620978 11377 raft_consensus.cc:3037] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.621711 11377 raft_consensus.cc:2454] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 2.
I20240501 19:56:38.622028 11447 leader_election.cc:302] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 8ad18cd54c6340fd92b874157875fa49, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:38.622220 16302 raft_consensus.cc:2781] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:38.622316 16302 raft_consensus.cc:686] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 2 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:38.622467 16302 consensus_queue.cc:237] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:38.622699 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169db26dfa3a4ec991c425c870e669eb" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:38.622794 13667 raft_consensus.cc:3037] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.623679 13667 raft_consensus.cc:2454] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 2.
I20240501 19:56:38.624557 14062 raft_consensus.cc:1203] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:38.626330 11100 catalog_manager.cc:5484] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 1 to 2, leader changed from ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 2 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.634244 11100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:38.640795 16332 tablet.cc:1722] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:38.641317 16334 tablet.cc:1722] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:38.642541 16333 tablet.cc:1722] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:38.646032 13669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be"
dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35384
I20240501 19:56:38.646168 13669 raft_consensus.cc:595] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.647161 16156 tablet_bootstrap.cc:492] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:38.647495 16156 tablet_bootstrap.cc:492] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:38.647755 16156 ts_tablet_manager.cc:1367] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.004s
I20240501 19:56:38.647163 11100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=7aa498ed1148430e8637a69bc29c296f] alter complete (version 3)
I20240501 19:56:38.648626 16156 raft_consensus.cc:348] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:38.648756 16156 raft_consensus.cc:729] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:38.649041 16156 consensus_queue.cc:260] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:38.649382 16156 ts_tablet_manager.cc:1398] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.649966 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-d505c415daa54005b8ed22bdc11ce135 received from {username='slave'} at 127.5.40.7:33205
I20240501 19:56:38.650056 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-d505c415daa54005b8ed22bdc11ce135 on tablet d505c415daa54005b8ed22bdc11ce135 with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:38.659795 13932 raft_consensus.cc:1203] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:38.684964 15930 consensus_queue.cc:1035] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.690191 16296 consensus_queue.cc:1035] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:38.721846 15815 raft_consensus.cc:979] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: : Instructing follower ef60ab93b49c4346af4f62d3b98558b8 to start an election
I20240501 19:56:38.721980 16188 raft_consensus.cc:1067] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Signalling peer ef60ab93b49c4346af4f62d3b98558b8 to start an election
I20240501 19:56:38.722602 11377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49"
dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
from {username='slave'} at 127.5.40.1:46917
I20240501 19:56:38.722756 11377 raft_consensus.cc:482] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:38.722817 11377 raft_consensus.cc:3037] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.723979 11377 raft_consensus.cc:504] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:38.724424 11377 leader_election.cc:288] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:38.725021 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:38.725154 11245 raft_consensus.cc:3032] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:38.725200 11245 raft_consensus.cc:729] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:38.725275 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:38.725394 13931 raft_consensus.cc:3037] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.725373 11245 consensus_queue.cc:260] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:38.725557 11245 raft_consensus.cc:3037] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.726399 13931 raft_consensus.cc:2454] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 2.
I20240501 19:56:38.726559 11245 raft_consensus.cc:2454] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 2.
I20240501 19:56:38.726828 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3176bda2ad4b9986cda48e4a03ce49" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:38.726944 11511 raft_consensus.cc:3037] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.727861 11313 leader_election.cc:302] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 6921cf2e41f64d66842885512da8de6c, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:38.727885 11511 raft_consensus.cc:2454] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 2.
I20240501 19:56:38.728026 15721 raft_consensus.cc:2781] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:38.728094 15721 raft_consensus.cc:686] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:38.728217 15721 consensus_queue.cc:237] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:38.730284 11097 catalog_manager.cc:5484] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 1 to 2, leader changed from 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 2 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.734364 16296 raft_consensus.cc:979] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: : Instructing follower 6921cf2e41f64d66842885512da8de6c to start an election
I20240501 19:56:38.734462 16296 raft_consensus.cc:1067] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Signalling peer 6921cf2e41f64d66842885512da8de6c to start an election
I20240501 19:56:38.734998 13931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be"
dest_uuid: "6921cf2e41f64d66842885512da8de6c"
from {username='slave'} at 127.5.40.5:50139
I20240501 19:56:38.735154 13931 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:38.735217 13931 raft_consensus.cc:3037] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.736320 13931 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:38.736904 13931 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 2 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:38.737212 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:38.737337 13800 raft_consensus.cc:3037] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.738180 13800 raft_consensus.cc:2454] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 2.
I20240501 19:56:38.738677 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:38.738818 11511 raft_consensus.cc:3037] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.739802 11511 raft_consensus.cc:2454] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 2.
I20240501 19:56:38.740150 13865 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:38.740340 16341 raft_consensus.cc:2781] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:38.740411 16341 raft_consensus.cc:686] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 2 LEADER]: Becoming Leader. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Running, Role: LEADER
I20240501 19:56:38.740561 16341 consensus_queue.cc:237] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:38.743099 11106 catalog_manager.cc:5484] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c reported cstate change: term changed from 1 to 2, leader changed from 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) to 6921cf2e41f64d66842885512da8de6c (127.5.40.7). New cstate: current_term: 2 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.747319 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07fd6b982437195ff42e5bf0c48be" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:38.747496 13669 raft_consensus.cc:3032] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:38.747542 13669 raft_consensus.cc:729] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:38.747709 13669 consensus_queue.cc:260] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:38.747896 13669 raft_consensus.cc:3037] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.749006 13669 raft_consensus.cc:2454] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 2.
I20240501 19:56:38.750748 11106 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:38.750886 11106 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:38.754122 11106 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet f65314238a604408a7f0ba3e3e62ced3
I20240501 19:56:38.754848 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet f65314238a604408a7f0ba3e3e62ced3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:38.755057 15732 tablet_replica.cc:330] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:38.755059 13648 tablet_service.cc:1514] Processing DeleteTablet for tablet f65314238a604408a7f0ba3e3e62ced3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:38.755189 15732 raft_consensus.cc:2229] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:38.755229 16284 tablet_replica.cc:330] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:38.755357 16284 raft_consensus.cc:2229] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.755398 15732 raft_consensus.cc:2258] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.755407 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet f65314238a604408a7f0ba3e3e62ced3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:38 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:38.755502 16284 raft_consensus.cc:2258] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.755596 15964 tablet_replica.cc:330] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:38.755713 15964 raft_consensus.cc:2229] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.755832 15964 raft_consensus.cc:2258] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.756086 16284 ts_tablet_manager.cc:1875] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.756103 15732 ts_tablet_manager.cc:1875] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.756515 15964 ts_tablet_manager.cc:1875] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:38.758219 16284 ts_tablet_manager.cc:1888] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:38.758219 15732 ts_tablet_manager.cc:1888] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:38.758285 16284 log.cc:1196] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/f65314238a604408a7f0ba3e3e62ced3
I20240501 19:56:38.758285 15732 log.cc:1196] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/f65314238a604408a7f0ba3e3e62ced3
I20240501 19:56:38.758525 16284 ts_tablet_manager.cc:1909] T f65314238a604408a7f0ba3e3e62ced3 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting consensus metadata
I20240501 19:56:38.758525 15732 ts_tablet_manager.cc:1909] T f65314238a604408a7f0ba3e3e62ced3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:38.758592 15964 ts_tablet_manager.cc:1888] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:38.758642 15964 log.cc:1196] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/f65314238a604408a7f0ba3e3e62ced3
I20240501 19:56:38.758855 15964 ts_tablet_manager.cc:1909] T f65314238a604408a7f0ba3e3e62ced3 P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:38.759524 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet f65314238a604408a7f0ba3e3e62ced3 (table rebalancer_dynamic_table_new_name [id=7aa498ed1148430e8637a69bc29c296f]) successfully deleted
I20240501 19:56:38.759707 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet f65314238a604408a7f0ba3e3e62ced3 (table rebalancer_dynamic_table_new_name [id=7aa498ed1148430e8637a69bc29c296f]) successfully deleted
I20240501 19:56:38.760154 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet f65314238a604408a7f0ba3e3e62ced3 (table rebalancer_dynamic_table_new_name [id=7aa498ed1148430e8637a69bc29c296f]) successfully deleted
I20240501 19:56:38.766927 11491 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:38.768627 11229 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:38.771827 11624 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:38.773541 13648 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:38.773985 13780 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:38.776032 11377 raft_consensus.cc:1203] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:38.791855 14043 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:38.797943 13800 raft_consensus.cc:1203] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:38.802381 11359 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:38.815366 11106 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:38.819881 13912 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:38.822324 13648 tablet_service.cc:1467] Processing CreateTablet for tablet 7677fcc04dc745cc9657d053fae31c36 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=88fd9227c05141ce860c32995c76a37d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.822589 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 7677fcc04dc745cc9657d053fae31c36 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=88fd9227c05141ce860c32995c76a37d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.822677 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7677fcc04dc745cc9657d053fae31c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.822671 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=88fd9227c05141ce860c32995c76a37d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.822865 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7677fcc04dc745cc9657d053fae31c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.822904 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.825240 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=88fd9227c05141ce860c32995c76a37d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.825537 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.825626 13912 tablet_service.cc:1467] Processing CreateTablet for tablet 7677fcc04dc745cc9657d053fae31c36 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=88fd9227c05141ce860c32995c76a37d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.825889 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7677fcc04dc745cc9657d053fae31c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.826140 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 6caecf09f4114855bddc470eb96bd5af (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=88fd9227c05141ce860c32995c76a37d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.826362 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caecf09f4114855bddc470eb96bd5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.827986 16538 tablet_bootstrap.cc:492] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:38.828161 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 6caecf09f4114855bddc470eb96bd5af (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=88fd9227c05141ce860c32995c76a37d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.828382 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caecf09f4114855bddc470eb96bd5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.828888 16538 tablet_bootstrap.cc:654] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.829905 16538 tablet_bootstrap.cc:492] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:38.829982 16538 ts_tablet_manager.cc:1367] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.830410 13911 tablet_service.cc:1467] Processing CreateTablet for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=88fd9227c05141ce860c32995c76a37d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.830449 16538 raft_consensus.cc:348] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:38.830505 16539 tablet_bootstrap.cc:492] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:38.830546 16538 raft_consensus.cc:374] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.830578 16538 raft_consensus.cc:729] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.830623 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.830706 16538 consensus_queue.cc:260] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:38.830996 16538 ts_tablet_manager.cc:1398] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:38.831341 16539 tablet_bootstrap.cc:654] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.832026 16536 tablet_bootstrap.cc:492] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:38.832029 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 6caecf09f4114855bddc470eb96bd5af (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=88fd9227c05141ce860c32995c76a37d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:38.832269 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caecf09f4114855bddc470eb96bd5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.832862 16536 tablet_bootstrap.cc:654] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.833173 16537 tablet_bootstrap.cc:492] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:38.833920 16539 tablet_bootstrap.cc:492] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:38.834069 16539 ts_tablet_manager.cc:1367] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:38.834110 16537 tablet_bootstrap.cc:654] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.834565 16539 raft_consensus.cc:348] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.834733 16539 raft_consensus.cc:374] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.834807 16539 raft_consensus.cc:729] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.834836 16535 tablet_bootstrap.cc:492] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:38.835167 16537 tablet_bootstrap.cc:492] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:38.835285 16537 ts_tablet_manager.cc:1367] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:38.835042 16539 consensus_queue.cc:260] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.835846 16537 raft_consensus.cc:348] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.835969 16537 raft_consensus.cc:374] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.835978 16535 tablet_bootstrap.cc:654] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.836012 16537 raft_consensus.cc:729] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.836153 16537 consensus_queue.cc:260] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.836495 16537 ts_tablet_manager.cc:1398] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.836632 16539 ts_tablet_manager.cc:1398] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:38.836660 16537 tablet_bootstrap.cc:492] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:38.836869 16539 tablet_bootstrap.cc:492] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:38.838169 16539 tablet_bootstrap.cc:654] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.839855 16539 tablet_bootstrap.cc:492] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:38.839995 16539 ts_tablet_manager.cc:1367] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:38.840473 11643 raft_consensus.cc:1203] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:38.840899 16539 raft_consensus.cc:348] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.841586 16539 raft_consensus.cc:374] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.841673 16539 raft_consensus.cc:729] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.841879 16539 consensus_queue.cc:260] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.842725 16539 ts_tablet_manager.cc:1398] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:38.843751 16539 tablet_bootstrap.cc:492] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:38.845407 16539 tablet_bootstrap.cc:654] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.846736 16539 tablet_bootstrap.cc:492] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:38.846833 16539 ts_tablet_manager.cc:1367] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:38.847191 16539 raft_consensus.cc:348] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:38.847267 16539 raft_consensus.cc:374] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.847286 16539 raft_consensus.cc:729] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.847388 16539 consensus_queue.cc:260] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:38.847651 16539 ts_tablet_manager.cc:1398] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.852128 16536 tablet_bootstrap.cc:492] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:38.852238 16536 ts_tablet_manager.cc:1367] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 19:56:38.852278 16535 tablet_bootstrap.cc:492] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:38.852360 16535 ts_tablet_manager.cc:1367] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240501 19:56:38.852726 16536 raft_consensus.cc:348] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:38.852769 16535 raft_consensus.cc:348] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.852838 16536 raft_consensus.cc:374] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.852851 16535 raft_consensus.cc:374] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.852867 16536 raft_consensus.cc:729] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.852893 16535 raft_consensus.cc:729] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.852999 16535 consensus_queue.cc:260] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.852999 16536 consensus_queue.cc:260] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:38.853274 16535 ts_tablet_manager.cc:1398] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:38.853296 16536 ts_tablet_manager.cc:1398] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.853466 16536 tablet_bootstrap.cc:492] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:38.854388 16536 tablet_bootstrap.cc:654] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.855407 16536 tablet_bootstrap.cc:492] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:38.855664 16537 tablet_bootstrap.cc:654] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.856904 16536 ts_tablet_manager.cc:1367] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:38.856839 11097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:38.857126 16537 tablet_bootstrap.cc:492] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:38.857316 16537 ts_tablet_manager.cc:1367] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 19:56:38.857553 16536 raft_consensus.cc:348] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.857714 16536 raft_consensus.cc:374] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.857789 16536 raft_consensus.cc:729] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.857770 16537 raft_consensus.cc:348] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.857865 16537 raft_consensus.cc:374] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.857897 16537 raft_consensus.cc:729] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.858026 16537 consensus_queue.cc:260] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.858083 16536 consensus_queue.cc:260] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:38.858384 16537 ts_tablet_manager.cc:1398] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.858486 16536 ts_tablet_manager.cc:1398] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.863425 13911 tablet_service.cc:1467] Processing CreateTablet for tablet d59e0502e63f4bff92b63db33a500dd7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a26c6589c9f44d36930670075d336a81]), partition=
I20240501 19:56:38.863642 14043 tablet_service.cc:1467] Processing CreateTablet for tablet d59e0502e63f4bff92b63db33a500dd7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a26c6589c9f44d36930670075d336a81]), partition=
I20240501 19:56:38.863731 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59e0502e63f4bff92b63db33a500dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.863848 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59e0502e63f4bff92b63db33a500dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.866035 16538 tablet_bootstrap.cc:492] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:38.866017 13648 tablet_service.cc:1467] Processing CreateTablet for tablet d59e0502e63f4bff92b63db33a500dd7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a26c6589c9f44d36930670075d336a81]), partition=
I20240501 19:56:38.866242 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59e0502e63f4bff92b63db33a500dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.866881 16538 tablet_bootstrap.cc:654] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.867496 16539 tablet_bootstrap.cc:492] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:38.868361 16539 tablet_bootstrap.cc:654] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.868362 16535 tablet_bootstrap.cc:492] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:38.869387 16535 tablet_bootstrap.cc:654] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.870585 16535 tablet_bootstrap.cc:492] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:38.870664 16535 ts_tablet_manager.cc:1367] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:38.871094 16535 raft_consensus.cc:348] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.871201 16535 raft_consensus.cc:374] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.871224 16535 raft_consensus.cc:729] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.871330 16535 consensus_queue.cc:260] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.871627 16535 ts_tablet_manager.cc:1398] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:38.874562 16539 tablet_bootstrap.cc:492] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:38.874657 16539 ts_tablet_manager.cc:1367] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:38.875101 16539 raft_consensus.cc:348] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.875209 16539 raft_consensus.cc:374] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.875241 16539 raft_consensus.cc:729] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.875362 16539 consensus_queue.cc:260] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.876170 16539 ts_tablet_manager.cc:1398] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:38.877465 16538 tablet_bootstrap.cc:492] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:38.877557 16538 ts_tablet_manager.cc:1367] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:38.878000 16538 raft_consensus.cc:348] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.878100 16538 raft_consensus.cc:374] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.878155 16538 raft_consensus.cc:729] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.878268 16538 consensus_queue.cc:260] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.878540 16538 ts_tablet_manager.cc:1398] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:38.901643 11511 consensus_queue.cc:237] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:38.903000 11643 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:38.903196 11445 consensus_peers.cc:485] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): Couldn't send request to peer 533908e517b24f77b2af8b5863e4ae5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fc7da07b6c5414bb305264962d0f218. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:38.903242 11377 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:38.903404 16302 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:38.903618 13437 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:38.905258 15765 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:38.905683 11377 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:38.905683 11643 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } } }
I20240501 19:56:38.907995 11099 catalog_manager.cc:5484] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.917184 14062 consensus_queue.cc:237] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:38.918713 11245 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.918756 13800 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:38.919028 13997 consensus_peers.cc:485] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): Couldn't send request to peer 8ad18cd54c6340fd92b874157875fa49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66bdddd3550b4ae8a97535e4747a5234. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:38.919102 16567 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:38.919301 16195 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:38.921326 16566 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:38.922866 11245 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:38.923202 13800 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:38.925976 11099 catalog_manager.cc:5484] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:38.934128 13800 consensus_queue.cc:237] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:38.935496 14062 raft_consensus.cc:1261] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.935604 13931 raft_consensus.cc:1261] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:38.935621 13735 consensus_peers.cc:485] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 -> Peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): Couldn't send request to peer 8ad18cd54c6340fd92b874157875fa49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c0ec896817246b19f753f94ff3f7af5. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:38.935916 14794 consensus_queue.cc:1035] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:38.936087 15507 consensus_queue.cc:1035] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:38.937820 16352 raft_consensus.cc:2932] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:38.938220 14062 raft_consensus.cc:2932] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:38.938251 13931 raft_consensus.cc:2932] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:38.940534 11097 catalog_manager.cc:5484] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.950479 14062 consensus_queue.cc:237] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:38.951891 13669 raft_consensus.cc:1261] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.951953 13800 raft_consensus.cc:1261] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:38.952123 13996 consensus_peers.cc:485] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): Couldn't send request to peer 08711534813d47ba85aaec0771ed5d0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a210b79583db455d86f2c68ee49aa4b7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:38.952354 16195 consensus_queue.cc:1035] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:38.952529 16566 consensus_queue.cc:1035] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:38.954159 16195 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:38.954600 13669 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:38.957438 11097 catalog_manager.cc:5484] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:38.961162 16341 raft_consensus.cc:482] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:38.961266 16341 raft_consensus.cc:504] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.961728 16341 leader_election.cc:288] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:38.962096 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59e0502e63f4bff92b63db33a500dd7" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:38.962229 13669 raft_consensus.cc:2454] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:38.962249 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59e0502e63f4bff92b63db33a500dd7" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:38.962364 14062 raft_consensus.cc:2454] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:38.962512 13865 leader_election.cc:302] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:38.962877 16341 raft_consensus.cc:2781] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:38.962932 16341 raft_consensus.cc:482] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:38.962980 16341 raft_consensus.cc:3037] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.963850 16341 raft_consensus.cc:504] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.964151 16341 leader_election.cc:288] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:38.964442 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59e0502e63f4bff92b63db33a500dd7" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:38.964548 13669 raft_consensus.cc:3037] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.964560 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59e0502e63f4bff92b63db33a500dd7" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:38.964658 14062 raft_consensus.cc:3037] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.965404 13669 raft_consensus.cc:2454] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:38.965602 14062 raft_consensus.cc:2454] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:38.965723 13865 leader_election.cc:302] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:38.966157 16341 raft_consensus.cc:2781] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.966233 16341 raft_consensus.cc:686] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Becoming Leader. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Running, Role: LEADER
I20240501 19:56:38.966353 16341 consensus_queue.cc:237] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:38.968711 11100 catalog_manager.cc:5484] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921cf2e41f64d66842885512da8de6c (127.5.40.7). New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.972571 13800 consensus_queue.cc:237] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
W20240501 19:56:38.974053 13736 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 -> Peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): Couldn't send request to peer ef60ab93b49c4346af4f62d3b98558b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8078e0cbffa943c98584224d05b0ece0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:38.975018 11245 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.975162 13669 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:38.975153 13800 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:38.975623 14794 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.975837 14935 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:38.978513 15507 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:38.978971 11245 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:38.979050 13669 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:38.981896 11097 catalog_manager.cc:5484] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 4 to 5, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.989672 13800 consensus_queue.cc:237] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:38.991039 14062 raft_consensus.cc:1261] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:38.991086 11245 raft_consensus.cc:1261] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:38.992244 13734 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af9c3a48f92e40f19d75f4151bb4c3e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:38.992476 14794 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:38.992668 16352 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:38.994513 14794 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:38.996004 14062 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:38.995983 11245 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:38.996948 11100 catalog_manager.cc:5484] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.006099 11511 consensus_queue.cc:237] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:39.007426 11377 raft_consensus.cc:1261] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:39.007566 11245 raft_consensus.cc:1261] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:39.007730 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eff09e5884f44e18a66583e8e0e73679. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:39.007970 13437 consensus_queue.cc:1035] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.008142 16302 consensus_queue.cc:1035] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.009739 13155 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.010144 11245 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.010545 11377 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.012845 11103 catalog_manager.cc:5484] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.025460 15721 raft_consensus.cc:1050] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:39.025846 15721 consensus_queue.cc:237] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:39.027338 11511 raft_consensus.cc:1261] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.027343 11643 raft_consensus.cc:1261] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.027685 13800 raft_consensus.cc:1261] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.028036 14473 consensus_queue.cc:1035] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.028239 16220 consensus_queue.cc:1035] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.028610 16619 consensus_queue.cc:1035] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.030145 16220 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.030548 11643 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.032310 11511 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.033128 11100 catalog_manager.cc:5484] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.034669 13800 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.037916 13931 raft_consensus.cc:1203] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:39.038110 11377 consensus_queue.cc:237] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:39.039429 13800 raft_consensus.cc:1261] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:39.039952 16619 consensus_queue.cc:1035] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.041139 11511 raft_consensus.cc:1261] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:39.041572 15721 consensus_queue.cc:1035] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.041944 16220 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.042353 13800 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.042459 11511 raft_consensus.cc:2932] T 61cadb3ab334435eaebe99fc3c9e496b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.043531 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61cadb3ab334435eaebe99fc3c9e496b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.044924 11100 catalog_manager.cc:5484] T 61cadb3ab334435eaebe99fc3c9e496b P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.047574 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 61cadb3ab334435eaebe99fc3c9e496b with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:39.048089 16623 tablet_replica.cc:330] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:39.048228 16623 raft_consensus.cc:2229] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.048394 16623 raft_consensus.cc:2258] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.048990 16623 ts_tablet_manager.cc:1875] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.050987 16623 ts_tablet_manager.cc:1888] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:39.051052 16623 log.cc:1196] T 61cadb3ab334435eaebe99fc3c9e496b P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/61cadb3ab334435eaebe99fc3c9e496b
I20240501 19:56:39.051528 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 61cadb3ab334435eaebe99fc3c9e496b (table rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]) successfully deleted
I20240501 19:56:39.058493 15806 raft_consensus.cc:482] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.058605 15806 raft_consensus.cc:504] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:39.058939 15806 leader_election.cc:288] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:39.059286 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad5cfa9b7a4a808eb9d0bb81f2a2c6" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:39.059413 13931 raft_consensus.cc:2454] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:39.059497 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad5cfa9b7a4a808eb9d0bb81f2a2c6" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:39.059659 14062 raft_consensus.cc:2454] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:39.059744 11577 leader_election.cc:302] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:39.059922 15806 raft_consensus.cc:2781] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.059989 15806 raft_consensus.cc:482] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.060034 15806 raft_consensus.cc:3037] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.060328 15081 raft_consensus.cc:482] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.060395 15081 raft_consensus.cc:504] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.060729 15081 leader_election.cc:288] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:39.060969 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caecf09f4114855bddc470eb96bd5af" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:39.061071 13931 raft_consensus.cc:2454] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:39.061087 15806 raft_consensus.cc:504] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:39.061133 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caecf09f4114855bddc470eb96bd5af" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:39.061251 13800 raft_consensus.cc:2454] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:39.061312 11577 leader_election.cc:302] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:39.061383 15806 leader_election.cc:288] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:39.061471 15806 raft_consensus.cc:2781] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.061540 15806 raft_consensus.cc:482] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.061574 15806 raft_consensus.cc:3037] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.061834 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad5cfa9b7a4a808eb9d0bb81f2a2c6" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:39.061933 13931 raft_consensus.cc:3037] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.062203 15806 raft_consensus.cc:504] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.062530 15806 leader_election.cc:288] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:39.062789 13931 raft_consensus.cc:2454] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:39.063014 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caecf09f4114855bddc470eb96bd5af" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:39.063117 13800 raft_consensus.cc:3037] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.063190 11577 leader_election.cc:302] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:39.063213 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad5cfa9b7a4a808eb9d0bb81f2a2c6" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:39.063321 14062 raft_consensus.cc:3037] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.063520 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caecf09f4114855bddc470eb96bd5af" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:39.063612 13931 raft_consensus.cc:3037] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.064128 13800 raft_consensus.cc:2454] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:39.064201 14062 raft_consensus.cc:2454] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:39.064440 13931 raft_consensus.cc:2454] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:39.064548 11580 leader_election.cc:302] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:39.064898 15081 raft_consensus.cc:2781] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.064998 15081 raft_consensus.cc:686] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:39.065143 15081 consensus_queue.cc:237] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:39.065800 15081 raft_consensus.cc:2781] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.065881 15081 raft_consensus.cc:686] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:39.065987 15081 consensus_queue.cc:237] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.069825 11103 catalog_manager.cc:5484] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.070164 11103 catalog_manager.cc:5484] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.073577 14794 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.073693 14794 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.074046 14794 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:39.074440 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7677fcc04dc745cc9657d053fae31c36" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:39.074450 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7677fcc04dc745cc9657d053fae31c36" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:39.074577 13931 raft_consensus.cc:2454] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:39.074604 13669 raft_consensus.cc:2454] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:39.074838 13734 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:39.075110 14794 raft_consensus.cc:2781] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.075162 14794 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.075196 14794 raft_consensus.cc:3037] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.076076 14794 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.076416 14794 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:39.076707 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7677fcc04dc745cc9657d053fae31c36" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:39.076743 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7677fcc04dc745cc9657d053fae31c36" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:39.076822 13931 raft_consensus.cc:3037] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.076835 13669 raft_consensus.cc:3037] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.077724 13931 raft_consensus.cc:2454] T 7677fcc04dc745cc9657d053fae31c36 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:39.077724 13669 raft_consensus.cc:2454] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:39.078032 13734 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:39.078200 14794 raft_consensus.cc:2781] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.078294 14794 raft_consensus.cc:686] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:39.078425 14794 consensus_queue.cc:237] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.079100 11103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:39.080209 11099 catalog_manager.cc:5484] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.084017 14063 raft_consensus.cc:1261] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.084362 13669 raft_consensus.cc:1261] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.084789 16643 ts_tablet_manager.cc:902] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:39.084935 16340 consensus_queue.cc:1035] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.085065 16341 consensus_queue.cc:1035] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.085304 16643 tablet_copy_client.cc:308] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:39.085742 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet 69facf86bba24f209c06afafc1ba8f7b from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:37783)
I20240501 19:56:39.085857 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet 69facf86bba24f209c06afafc1ba8f7b from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:37783: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-69facf86bba24f209c06afafc1ba8f7b
I20240501 19:56:39.086845 11664 tablet_copy_source_session.cc:205] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.087989 16643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69facf86bba24f209c06afafc1ba8f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.087977 11099 catalog_manager.cc:5576] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501) does not have the latest schema for tablet d59e0502e63f4bff92b63db33a500dd7 (table rebalancer_dynamic_table [id=a26c6589c9f44d36930670075d336a81]). Expected version 1 got 0
I20240501 19:56:39.088204 16333 tablet.cc:1722] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:39.088527 16333 tablet.cc:1620] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:39.089932 16334 tablet.cc:1722] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:39.090217 16334 tablet.cc:1620] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:39.090590 16643 tablet_copy_client.cc:783] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.090731 16643 tablet_copy_client.cc:647] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.093153 11099 catalog_manager.cc:5576] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789) does not have the latest schema for tablet d59e0502e63f4bff92b63db33a500dd7 (table rebalancer_dynamic_table [id=a26c6589c9f44d36930670075d336a81]). Expected version 1 got 0
I20240501 19:56:39.093592 16643 tablet_copy_client.cc:523] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.094024 11100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=a26c6589c9f44d36930670075d336a81] alter complete (version 1)
I20240501 19:56:39.094811 16643 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:39.098793 16657 tablet.cc:1722] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:39.099306 16657 tablet.cc:1620] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:39.100209 11099 catalog_manager.cc:5576] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789) does not have the latest schema for tablet d59e0502e63f4bff92b63db33a500dd7 (table rebalancer_dynamic_table [id=a26c6589c9f44d36930670075d336a81]). Expected version 1 got 0
I20240501 19:56:39.110014 13931 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:39.110576 16619 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.119203 16643 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.119556 16643 tablet_bootstrap.cc:492] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:39.119735 16643 ts_tablet_manager.cc:1367] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.003s
I20240501 19:56:39.120152 16643 raft_consensus.cc:348] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:39.120265 16643 raft_consensus.cc:729] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:39.120360 16643 consensus_queue.cc:260] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:39.120661 16643 ts_tablet_manager.cc:1398] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:39.121136 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-69facf86bba24f209c06afafc1ba8f7b received from {username='slave'} at 127.5.40.6:37783
I20240501 19:56:39.121210 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-69facf86bba24f209c06afafc1ba8f7b on tablet 69facf86bba24f209c06afafc1ba8f7b with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:39.140548 11511 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:39.141007 16619 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.144657 11245 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:39.145442 16178 raft_consensus.cc:1050] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:39.145637 11377 consensus_queue.cc:237] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:39.145839 16178 consensus_queue.cc:237] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:39.147217 13931 raft_consensus.cc:1261] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.147596 15806 consensus_queue.cc:1035] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.147648 11511 raft_consensus.cc:1261] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.148029 11245 raft_consensus.cc:1261] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.148607 15081 consensus_queue.cc:1035] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.148761 15806 consensus_queue.cc:1035] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.149741 15765 raft_consensus.cc:1050] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:39.150110 15765 consensus_queue.cc:237] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:39.150203 11508 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:56:39.150208 13932 raft_consensus.cc:1261] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:56:39.151194 15721 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:39.151684 11643 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:39.152097 14473 consensus_queue.cc:1035] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:39.152550 14063 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:39.152916 13669 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:39.153420 13155 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:39.153415 15721 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.153760 16178 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.154076 13155 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:39.155210 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f3176bda2ad4b9986cda48e4a03ce49 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.155597 13931 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.155804 11511 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.157109 11099 catalog_manager.cc:5484] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 5 to 7, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New cstate: current_term: 2 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.157538 13155 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:39.157495 11103 catalog_manager.cc:5484] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.158986 13932 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.159992 13155 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:39.161633 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3176bda2ad4b9986cda48e4a03ce49 with delete_type TABLET_DATA_TOMBSTONED (TS 10ee60ead7b94c93ab749ec16a04b0dc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:39.161669 11643 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:39.162429 13437 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.162467 11097 catalog_manager.cc:5484] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.163120 11644 consensus_queue.cc:237] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:39.163008 11245 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.163307 13669 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:39.163936 11508 raft_consensus.cc:2932] T 8f3176bda2ad4b9986cda48e4a03ce49 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.165903 14063 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:39.166414 11508 raft_consensus.cc:1261] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:39.166738 16178 consensus_queue.cc:1035] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.168282 16178 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.168604 11643 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:39.169978 13932 raft_consensus.cc:1261] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:39.170464 16694 consensus_queue.cc:1035] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.170594 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1772073172764f8db95c597f8da14f12 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.171940 13932 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.171397 11103 catalog_manager.cc:5484] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.173955 16702 tablet_replica.cc:330] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:39.174146 16702 raft_consensus.cc:2229] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.174324 16702 raft_consensus.cc:2258] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.175074 16702 ts_tablet_manager.cc:1875] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.176218 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet 1772073172764f8db95c597f8da14f12 with delete_type TABLET_DATA_TOMBSTONED (TS 10ee60ead7b94c93ab749ec16a04b0dc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:39.176615 11508 raft_consensus.cc:2932] T 1772073172764f8db95c597f8da14f12 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.178329 11511 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e63472cfff449089d88d18b9e495912"
dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50198
I20240501 19:56:39.178468 11511 raft_consensus.cc:595] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:39.180326 16702 ts_tablet_manager.cc:1888] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:39.180398 16702 log.cc:1196] T 8f3176bda2ad4b9986cda48e4a03ce49 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f3176bda2ad4b9986cda48e4a03ce49
I20240501 19:56:39.181080 16702 tablet_replica.cc:330] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:39.181170 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 8f3176bda2ad4b9986cda48e4a03ce49 (table rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]) successfully deleted
I20240501 19:56:39.181370 16702 raft_consensus.cc:2229] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.181566 16702 raft_consensus.cc:2258] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.182323 16702 ts_tablet_manager.cc:1875] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.183219 11377 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:39.183629 15765 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.185050 16702 ts_tablet_manager.cc:1888] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:39.187682 16702 log.cc:1196] T 1772073172764f8db95c597f8da14f12 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/1772073172764f8db95c597f8da14f12
I20240501 19:56:39.188930 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 1772073172764f8db95c597f8da14f12 (table rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]) successfully deleted
I20240501 19:56:39.189051 13669 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:39.189476 15765 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.197774 11103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:39.203263 16333 tablet.cc:1722] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:39.203605 16333 tablet.cc:1620] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:39.204155 16657 tablet.cc:1722] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:39.204433 16657 tablet.cc:1620] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:39.204567 16334 tablet.cc:1722] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:39.204854 16334 tablet.cc:1620] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:39.209112 11103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=a26c6589c9f44d36930670075d336a81] alter complete (version 2)
I20240501 19:56:39.220595 13800 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 2 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:39.221246 16340 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.236651 11377 consensus_queue.cc:237] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:39.237970 11511 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:39.238025 11643 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:39.238108 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6c1d3814e364e718449c3e4379b31ed. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:39.238373 14473 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.238725 16619 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.240170 15721 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:39.240831 11643 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:39.240831 11511 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:39.243547 11106 catalog_manager.cc:5484] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:39.255205 15931 raft_consensus.cc:1050] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: attempting to promote NON_VOTER 8ad18cd54c6340fd92b874157875fa49 to VOTER
I20240501 19:56:39.255704 15931 consensus_queue.cc:237] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:39.257423 11245 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:39.257419 14063 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:39.257784 11643 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:39.258102 16296 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:39.258271 16295 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:39.258517 16754 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:39.260646 16295 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:39.261673 14063 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:39.261657 11643 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:39.264532 11100 catalog_manager.cc:5484] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:39.264614 11245 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:39.279543 11511 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:39.280776 16340 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.281227 14794 raft_consensus.cc:1050] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7: attempting to promote NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 to VOTER
I20240501 19:56:39.281656 14794 consensus_queue.cc:237] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:39.282967 11380 raft_consensus.cc:1261] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:39.283198 11245 raft_consensus.cc:1261] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:39.283706 13932 raft_consensus.cc:1261] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:39.284706 15507 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:39.284965 13669 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:39.285358 16341 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.286149 13932 consensus_queue.cc:237] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:39.287602 11511 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:39.288035 16341 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.289155 13800 raft_consensus.cc:1261] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 2 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:39.289489 15507 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:39.289613 16341 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:39.289793 15507 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:39.291021 16341 consensus_queue.cc:1035] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.291541 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dcb07fd6b982437195ff42e5bf0c48be with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.292605 11106 catalog_manager.cc:5484] T dcb07fd6b982437195ff42e5bf0c48be P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 3 to 5, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New cstate: current_term: 2 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.292650 13800 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:39.294365 15507 raft_consensus.cc:2932] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:39.294529 11508 raft_consensus.cc:2932] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:39.296144 11380 raft_consensus.cc:2932] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:39.296303 13648 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb07fd6b982437195ff42e5bf0c48be with delete_type TABLET_DATA_TOMBSTONED (TS 8125d8ad6d0d44caae9c5c90c90df96b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:39.296741 16778 tablet_replica.cc:330] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:39.296929 16778 raft_consensus.cc:2229] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.296870 11245 raft_consensus.cc:2932] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:39.297019 16778 raft_consensus.cc:2258] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.297564 16778 ts_tablet_manager.cc:1875] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.299693 16778 ts_tablet_manager.cc:1888] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 19:56:39.299762 16778 log.cc:1196] T dcb07fd6b982437195ff42e5bf0c48be P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/dcb07fd6b982437195ff42e5bf0c48be
I20240501 19:56:39.299327 11106 catalog_manager.cc:5484] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 2 to 3, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.301079 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet dcb07fd6b982437195ff42e5bf0c48be (table rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]) successfully deleted
I20240501 19:56:39.301834 14473 raft_consensus.cc:1050] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:39.302258 14473 consensus_queue.cc:237] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:39.302533 13800 consensus_queue.cc:237] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:39.303669 11245 raft_consensus.cc:1261] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:39.303725 11643 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:39.304183 11380 raft_consensus.cc:1261] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:39.304224 14063 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:39.304502 14935 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.304579 15721 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:39.304756 16220 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:39.305228 14935 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.306391 13800 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:39.306766 15721 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:39.307708 16220 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.308187 14935 raft_consensus.cc:2932] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:39.308171 14063 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.308789 11380 raft_consensus.cc:2932] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:39.310134 11092 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87aabbb9dc2440f28a3b6944dea38fe7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.310514 11643 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.311035 11106 catalog_manager.cc:5484] T 87aabbb9dc2440f28a3b6944dea38fe7 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.312175 11245 raft_consensus.cc:2932] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:39.313311 13911 tablet_service.cc:1514] Processing DeleteTablet for tablet 87aabbb9dc2440f28a3b6944dea38fe7 with delete_type TABLET_DATA_TOMBSTONED (TS 6921cf2e41f64d66842885512da8de6c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:39.313517 11103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:39.313913 16787 tablet_replica.cc:330] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:39.314043 16787 raft_consensus.cc:2229] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.314150 16787 pending_rounds.cc:62] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: Trying to abort 1 pending ops.
I20240501 19:56:39.314402 11106 catalog_manager.cc:5484] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.316141 13800 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.320993 16333 tablet.cc:1722] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:39.321503 16657 tablet.cc:1722] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:39.314209 16787 pending_rounds.cc:68] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022974563456552960 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "87aabbb9dc2440f28a3b6944dea38fe7" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } } }
I20240501 19:56:39.322477 16787 raft_consensus.cc:2894] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 19:56:39.322548 16787 raft_consensus.cc:2866] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:39.322594 16787 raft_consensus.cc:2258] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.323089 16787 ts_tablet_manager.cc:1875] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.324975 16334 tablet.cc:1722] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:39.325412 11100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=a26c6589c9f44d36930670075d336a81] alter complete (version 3)
I20240501 19:56:39.325791 16787 ts_tablet_manager.cc:1888] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:39.325863 16787 log.cc:1196] T 87aabbb9dc2440f28a3b6944dea38fe7 P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/87aabbb9dc2440f28a3b6944dea38fe7
I20240501 19:56:39.326431 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet 87aabbb9dc2440f28a3b6944dea38fe7 (table rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]) successfully deleted
I20240501 19:56:39.330747 11377 consensus_queue.cc:237] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:39.331988 13800 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:39.332079 14063 raft_consensus.cc:1261] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:39.332727 14473 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:39.332916 15721 consensus_queue.cc:1035] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:39.334623 16619 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.335028 13800 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.335214 14063 raft_consensus.cc:2932] T cebd15701ee44b11a2f8804740fb4299 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.336478 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cebd15701ee44b11a2f8804740fb4299 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.337806 11103 catalog_manager.cc:5484] T cebd15701ee44b11a2f8804740fb4299 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.340312 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet cebd15701ee44b11a2f8804740fb4299 with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:39.340698 16623 tablet_replica.cc:330] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:39.340842 16623 raft_consensus.cc:2229] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.341017 16623 raft_consensus.cc:2258] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.341730 16623 ts_tablet_manager.cc:1875] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.343880 16623 ts_tablet_manager.cc:1888] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:39.344009 16623 log.cc:1196] T cebd15701ee44b11a2f8804740fb4299 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/cebd15701ee44b11a2f8804740fb4299
I20240501 19:56:39.344456 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet cebd15701ee44b11a2f8804740fb4299 (table rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]) successfully deleted
I20240501 19:56:39.370801 16818 ts_tablet_manager.cc:902] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: Initiating tablet copy from peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:39.371371 16818 tablet_copy_client.cc:308] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.8:34789
I20240501 19:56:39.372076 14083 tablet_copy_service.cc:140] P 533908e517b24f77b2af8b5863e4ae5a: Received BeginTabletCopySession request for tablet a210b79583db455d86f2c68ee49aa4b7 from peer 08711534813d47ba85aaec0771ed5d0f ({username='slave'} at 127.5.40.3:49777)
I20240501 19:56:39.372210 14083 tablet_copy_service.cc:161] P 533908e517b24f77b2af8b5863e4ae5a: Beginning new tablet copy session on tablet a210b79583db455d86f2c68ee49aa4b7 from peer 08711534813d47ba85aaec0771ed5d0f at {username='slave'} at 127.5.40.3:49777: session id = 08711534813d47ba85aaec0771ed5d0f-a210b79583db455d86f2c68ee49aa4b7
I20240501 19:56:39.373418 14083 tablet_copy_source_session.cc:205] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.374442 16818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a210b79583db455d86f2c68ee49aa4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.376580 16818 tablet_copy_client.cc:783] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.376731 16818 tablet_copy_client.cc:647] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.378855 16818 tablet_copy_client.cc:523] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.380256 16818 tablet_bootstrap.cc:492] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:39.384337 16818 tablet_bootstrap.cc:492] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.384550 16818 tablet_bootstrap.cc:492] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap complete.
I20240501 19:56:39.384693 16818 ts_tablet_manager.cc:1367] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:39.385033 16818 raft_consensus.cc:348] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:39.385135 16818 raft_consensus.cc:729] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: LEARNER
I20240501 19:56:39.385226 16818 consensus_queue.cc:260] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:39.385445 16818 ts_tablet_manager.cc:1398] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:39.386078 14083 tablet_copy_service.cc:342] P 533908e517b24f77b2af8b5863e4ae5a: Request end of tablet copy session 08711534813d47ba85aaec0771ed5d0f-a210b79583db455d86f2c68ee49aa4b7 received from {username='slave'} at 127.5.40.3:49777
I20240501 19:56:39.386176 14083 tablet_copy_service.cc:433] P 533908e517b24f77b2af8b5863e4ae5a: ending tablet copy session 08711534813d47ba85aaec0771ed5d0f-a210b79583db455d86f2c68ee49aa4b7 on tablet a210b79583db455d86f2c68ee49aa4b7 with peer 08711534813d47ba85aaec0771ed5d0f
I20240501 19:56:39.395776 13669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96511455168148b7b619670f635d3a3e"
dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35422
I20240501 19:56:39.395910 13669 raft_consensus.cc:595] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:39.406234 11508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e63472cfff449089d88d18b9e495912"
dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50256
I20240501 19:56:39.406365 11508 raft_consensus.cc:595] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:39.425148 13669 consensus_queue.cc:237] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:39.426481 11508 raft_consensus.cc:1261] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.426544 13932 raft_consensus.cc:1261] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:39.426759 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): Couldn't send request to peer 10ee60ead7b94c93ab749ec16a04b0dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 811d018fde314227a378f2d530e0a924. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:39.426952 15931 consensus_queue.cc:1035] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:39.427155 16296 consensus_queue.cc:1035] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:39.428735 16295 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:39.429216 11508 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:39.429225 13932 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:39.431679 11103 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:39.431815 11103 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:39.432029 11099 catalog_manager.cc:5484] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:39.435230 11103 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet d59e0502e63f4bff92b63db33a500dd7
I20240501 19:56:39.435822 13911 tablet_service.cc:1514] Processing DeleteTablet for tablet d59e0502e63f4bff92b63db33a500dd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:39 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:39.435927 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet d59e0502e63f4bff92b63db33a500dd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:39 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:39.436007 16787 tablet_replica.cc:330] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:39.436134 16787 raft_consensus.cc:2229] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:39.436226 16865 tablet_replica.cc:330] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:39.436213 13648 tablet_service.cc:1514] Processing DeleteTablet for tablet d59e0502e63f4bff92b63db33a500dd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:39 UTC) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:39.436342 16865 raft_consensus.cc:2229] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.436336 16787 raft_consensus.cc:2258] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.436367 16778 tablet_replica.cc:330] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:39.436437 16865 raft_consensus.cc:2258] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.436493 16778 raft_consensus.cc:2229] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.436596 16778 raft_consensus.cc:2258] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.436969 16865 ts_tablet_manager.cc:1875] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:39.436975 16787 ts_tablet_manager.cc:1875] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:39.439070 16865 ts_tablet_manager.cc:1888] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:39.439070 16787 ts_tablet_manager.cc:1888] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:39.439148 16865 log.cc:1196] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/d59e0502e63f4bff92b63db33a500dd7
I20240501 19:56:39.439148 16787 log.cc:1196] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/d59e0502e63f4bff92b63db33a500dd7
I20240501 19:56:39.439400 16865 ts_tablet_manager.cc:1909] T d59e0502e63f4bff92b63db33a500dd7 P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:39.439400 16787 ts_tablet_manager.cc:1909] T d59e0502e63f4bff92b63db33a500dd7 P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:39.439953 16778 ts_tablet_manager.cc:1875] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:39.440163 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet d59e0502e63f4bff92b63db33a500dd7 (table rebalancer_dynamic_table_new_name [id=a26c6589c9f44d36930670075d336a81]) successfully deleted
I20240501 19:56:39.440325 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet d59e0502e63f4bff92b63db33a500dd7 (table rebalancer_dynamic_table_new_name [id=a26c6589c9f44d36930670075d336a81]) successfully deleted
I20240501 19:56:39.441989 16778 ts_tablet_manager.cc:1888] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:39.442070 16778 log.cc:1196] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/d59e0502e63f4bff92b63db33a500dd7
I20240501 19:56:39.442312 16778 ts_tablet_manager.cc:1909] T d59e0502e63f4bff92b63db33a500dd7 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting consensus metadata
I20240501 19:56:39.442937 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet d59e0502e63f4bff92b63db33a500dd7 (table rebalancer_dynamic_table_new_name [id=a26c6589c9f44d36930670075d336a81]) successfully deleted
I20240501 19:56:39.445358 11377 consensus_queue.cc:237] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:39.446849 14063 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:39.446945 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01a10926bce14cf1bd12f21287e08c89. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:39.447223 11643 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:39.447224 14473 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:39.447593 14473 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:39.449015 16220 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.449405 14063 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.449744 11643 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.452045 11097 catalog_manager.cc:5484] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.455075 16178 consensus_queue.cc:1035] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.455265 16694 consensus_queue.cc:1035] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.459544 16178 consensus_queue.cc:1035] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.459816 16178 consensus_queue.cc:1035] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.466461 11508 consensus_queue.cc:237] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:39.467795 11377 raft_consensus.cc:1261] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:39.467921 11445 consensus_peers.cc:485] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f594c53151b4d00a1784b6366e2814a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:39.467967 11245 raft_consensus.cc:1261] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:39.468231 15765 consensus_queue.cc:1035] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.468394 13155 consensus_queue.cc:1035] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:39.468964 14794 consensus_queue.cc:1035] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.470326 15765 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:39.472965 11100 catalog_manager.cc:5484] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.473335 11377 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:39.473513 11245 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:39.477864 16352 consensus_queue.cc:1035] T 7677fcc04dc745cc9657d053fae31c36 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.478066 16883 ts_tablet_manager.cc:902] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:39.478961 16883 tablet_copy_client.cc:308] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:39.479377 16892 ts_tablet_manager.cc:902] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: Initiating tablet copy from peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:39.479638 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 1fc7da07b6c5414bb305264962d0f218 from peer 533908e517b24f77b2af8b5863e4ae5a ({username='slave'} at 127.5.40.8:54331)
I20240501 19:56:39.479732 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 1fc7da07b6c5414bb305264962d0f218 from peer 533908e517b24f77b2af8b5863e4ae5a at {username='slave'} at 127.5.40.8:54331: session id = 533908e517b24f77b2af8b5863e4ae5a-1fc7da07b6c5414bb305264962d0f218
I20240501 19:56:39.479817 16882 ts_tablet_manager.cc:902] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: Initiating tablet copy from peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:39.479998 16892 tablet_copy_client.cc:308] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.6:46331
I20240501 19:56:39.480273 16882 tablet_copy_client.cc:308] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.8:34789
I20240501 19:56:39.480628 13821 tablet_copy_service.cc:140] P 96fc18dbb5fd41afb9bebf805e8d33a7: Received BeginTabletCopySession request for tablet 7c0ec896817246b19f753f94ff3f7af5 from peer 8ad18cd54c6340fd92b874157875fa49 ({username='slave'} at 127.5.40.4:56917)
I20240501 19:56:39.480715 11533 tablet_copy_source_session.cc:205] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.480727 13821 tablet_copy_service.cc:161] P 96fc18dbb5fd41afb9bebf805e8d33a7: Beginning new tablet copy session on tablet 7c0ec896817246b19f753f94ff3f7af5 from peer 8ad18cd54c6340fd92b874157875fa49 at {username='slave'} at 127.5.40.4:56917: session id = 8ad18cd54c6340fd92b874157875fa49-7c0ec896817246b19f753f94ff3f7af5
I20240501 19:56:39.481055 14083 tablet_copy_service.cc:140] P 533908e517b24f77b2af8b5863e4ae5a: Received BeginTabletCopySession request for tablet 66bdddd3550b4ae8a97535e4747a5234 from peer 8ad18cd54c6340fd92b874157875fa49 ({username='slave'} at 127.5.40.4:33773)
I20240501 19:56:39.481137 14083 tablet_copy_service.cc:161] P 533908e517b24f77b2af8b5863e4ae5a: Beginning new tablet copy session on tablet 66bdddd3550b4ae8a97535e4747a5234 from peer 8ad18cd54c6340fd92b874157875fa49 at {username='slave'} at 127.5.40.4:33773: session id = 8ad18cd54c6340fd92b874157875fa49-66bdddd3550b4ae8a97535e4747a5234
I20240501 19:56:39.481576 13821 tablet_copy_source_session.cc:205] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.481637 16883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7da07b6c5414bb305264962d0f218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.482019 14083 tablet_copy_source_session.cc:205] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.484083 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bdddd3550b4ae8a97535e4747a5234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.484110 13800 raft_consensus.cc:1203] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:39.485033 16883 tablet_copy_client.cc:783] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.485174 16892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0ec896817246b19f753f94ff3f7af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.485294 16883 tablet_copy_client.cc:647] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.486485 16883 tablet_copy_client.cc:523] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.487620 16892 tablet_copy_client.cc:783] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.487650 16883 tablet_bootstrap.cc:492] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:39.487604 16882 tablet_copy_client.cc:783] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.487785 16892 tablet_copy_client.cc:647] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.487915 16882 tablet_copy_client.cc:647] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.488735 16892 tablet_copy_client.cc:523] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.488844 16902 ts_tablet_manager.cc:902] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: Initiating tablet copy from peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:39.489888 16892 tablet_bootstrap.cc:492] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:39.490381 16902 tablet_copy_client.cc:308] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.6:46331
I20240501 19:56:39.490821 13821 tablet_copy_service.cc:140] P 96fc18dbb5fd41afb9bebf805e8d33a7: Received BeginTabletCopySession request for tablet 8078e0cbffa943c98584224d05b0ece0 from peer ef60ab93b49c4346af4f62d3b98558b8 ({username='slave'} at 127.5.40.2:37395)
I20240501 19:56:39.490912 13821 tablet_copy_service.cc:161] P 96fc18dbb5fd41afb9bebf805e8d33a7: Beginning new tablet copy session on tablet 8078e0cbffa943c98584224d05b0ece0 from peer ef60ab93b49c4346af4f62d3b98558b8 at {username='slave'} at 127.5.40.2:37395: session id = ef60ab93b49c4346af4f62d3b98558b8-8078e0cbffa943c98584224d05b0ece0
I20240501 19:56:39.491777 16882 tablet_copy_client.cc:523] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.491808 13821 tablet_copy_source_session.cc:205] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.492663 16902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8078e0cbffa943c98584224d05b0ece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.494072 16882 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:39.494915 16902 tablet_copy_client.cc:783] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.495033 16902 tablet_copy_client.cc:647] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.496547 16902 tablet_copy_client.cc:523] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.497057 16892 tablet_bootstrap.cc:492] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.497259 16892 tablet_bootstrap.cc:492] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap complete.
I20240501 19:56:39.497404 16892 ts_tablet_manager.cc:1367] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 19:56:39.497813 16902 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:39.497802 16892 raft_consensus.cc:348] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:39.497952 16892 raft_consensus.cc:729] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: LEARNER
I20240501 19:56:39.498060 16892 consensus_queue.cc:260] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:39.498426 16892 ts_tablet_manager.cc:1398] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.500258 13821 tablet_copy_service.cc:342] P 96fc18dbb5fd41afb9bebf805e8d33a7: Request end of tablet copy session 8ad18cd54c6340fd92b874157875fa49-7c0ec896817246b19f753f94ff3f7af5 received from {username='slave'} at 127.5.40.4:56917
I20240501 19:56:39.500341 13821 tablet_copy_service.cc:433] P 96fc18dbb5fd41afb9bebf805e8d33a7: ending tablet copy session 8ad18cd54c6340fd92b874157875fa49-7c0ec896817246b19f753f94ff3f7af5 on tablet 7c0ec896817246b19f753f94ff3f7af5 with peer 8ad18cd54c6340fd92b874157875fa49
I20240501 19:56:39.502247 16882 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.502501 16882 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap complete.
I20240501 19:56:39.502662 16882 ts_tablet_manager.cc:1367] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 19:56:39.503203 16882 raft_consensus.cc:348] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:39.503340 16882 raft_consensus.cc:729] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: LEARNER
I20240501 19:56:39.503486 16882 consensus_queue.cc:260] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:39.503865 16882 ts_tablet_manager.cc:1398] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.504483 14083 tablet_copy_service.cc:342] P 533908e517b24f77b2af8b5863e4ae5a: Request end of tablet copy session 8ad18cd54c6340fd92b874157875fa49-66bdddd3550b4ae8a97535e4747a5234 received from {username='slave'} at 127.5.40.4:33773
I20240501 19:56:39.504568 14083 tablet_copy_service.cc:433] P 533908e517b24f77b2af8b5863e4ae5a: ending tablet copy session 8ad18cd54c6340fd92b874157875fa49-66bdddd3550b4ae8a97535e4747a5234 on tablet 66bdddd3550b4ae8a97535e4747a5234 with peer 8ad18cd54c6340fd92b874157875fa49
I20240501 19:56:39.505599 16902 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.505832 16902 tablet_bootstrap.cc:492] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap complete.
I20240501 19:56:39.505995 16902 ts_tablet_manager.cc:1367] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 19:56:39.506403 16902 raft_consensus.cc:348] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:39.506532 16902 raft_consensus.cc:729] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: LEARNER
I20240501 19:56:39.506629 16916 ts_tablet_manager.cc:902] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:39.506631 16902 consensus_queue.cc:260] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:39.507112 16916 tablet_copy_client.cc:308] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:39.507629 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet eff09e5884f44e18a66583e8e0e73679 from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:41129)
I20240501 19:56:39.507722 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet eff09e5884f44e18a66583e8e0e73679 from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:41129: session id = 6921cf2e41f64d66842885512da8de6c-eff09e5884f44e18a66583e8e0e73679
I20240501 19:56:39.508563 16902 ts_tablet_manager.cc:1398] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:39.508663 11533 tablet_copy_source_session.cc:205] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.509487 13821 tablet_copy_service.cc:342] P 96fc18dbb5fd41afb9bebf805e8d33a7: Request end of tablet copy session ef60ab93b49c4346af4f62d3b98558b8-8078e0cbffa943c98584224d05b0ece0 received from {username='slave'} at 127.5.40.2:37395
I20240501 19:56:39.509562 13821 tablet_copy_service.cc:433] P 96fc18dbb5fd41afb9bebf805e8d33a7: ending tablet copy session ef60ab93b49c4346af4f62d3b98558b8-8078e0cbffa943c98584224d05b0ece0 on tablet 8078e0cbffa943c98584224d05b0ece0 with peer ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:39.509896 16916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff09e5884f44e18a66583e8e0e73679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.512183 16916 tablet_copy_client.cc:783] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.512331 16916 tablet_copy_client.cc:647] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.514120 16883 tablet_bootstrap.cc:492] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.514529 16883 tablet_bootstrap.cc:492] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap complete.
I20240501 19:56:39.514767 16883 ts_tablet_manager.cc:1367] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.007s
I20240501 19:56:39.516892 16883 raft_consensus.cc:348] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:39.517046 16883 raft_consensus.cc:729] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: LEARNER
I20240501 19:56:39.517189 16883 consensus_queue.cc:260] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: NON_VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: true } }
I20240501 19:56:39.517414 16916 tablet_copy_client.cc:523] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.517535 16883 ts_tablet_manager.cc:1398] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:39.518046 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 533908e517b24f77b2af8b5863e4ae5a-1fc7da07b6c5414bb305264962d0f218 received from {username='slave'} at 127.5.40.8:54331
I20240501 19:56:39.518129 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 533908e517b24f77b2af8b5863e4ae5a-1fc7da07b6c5414bb305264962d0f218 on tablet 1fc7da07b6c5414bb305264962d0f218 with peer 533908e517b24f77b2af8b5863e4ae5a
I20240501 19:56:39.518751 16916 tablet_bootstrap.cc:492] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:39.537103 11103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:39.543936 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 546565eab21645c2add5eb6726f9e83d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e1afee203712425fa19eb1ee8ec389ea]), partition=
I20240501 19:56:39.544039 13648 tablet_service.cc:1467] Processing CreateTablet for tablet 546565eab21645c2add5eb6726f9e83d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e1afee203712425fa19eb1ee8ec389ea]), partition=
I20240501 19:56:39.544035 13911 tablet_service.cc:1467] Processing CreateTablet for tablet 546565eab21645c2add5eb6726f9e83d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e1afee203712425fa19eb1ee8ec389ea]), partition=
I20240501 19:56:39.544204 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546565eab21645c2add5eb6726f9e83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.544260 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546565eab21645c2add5eb6726f9e83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.544260 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 546565eab21645c2add5eb6726f9e83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.545965 16938 tablet_bootstrap.cc:492] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:39.543808 16916 tablet_bootstrap.cc:492] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.546674 16916 tablet_bootstrap.cc:492] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:39.546937 16916 ts_tablet_manager.cc:1367] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.028s user 0.025s sys 0.000s
I20240501 19:56:39.547451 16916 raft_consensus.cc:348] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:39.547650 16916 raft_consensus.cc:729] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:39.547801 16938 tablet_bootstrap.cc:654] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.547804 16916 consensus_queue.cc:260] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:39.548781 16916 ts_tablet_manager.cc:1398] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:39.549273 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-eff09e5884f44e18a66583e8e0e73679 received from {username='slave'} at 127.5.40.7:41129
I20240501 19:56:39.549357 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-eff09e5884f44e18a66583e8e0e73679 on tablet eff09e5884f44e18a66583e8e0e73679 with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:39.550135 16938 tablet_bootstrap.cc:492] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:39.550213 16938 ts_tablet_manager.cc:1367] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:39.550604 16938 raft_consensus.cc:348] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:39.550704 16938 raft_consensus.cc:374] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.550731 16938 raft_consensus.cc:729] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.550841 16938 consensus_queue.cc:260] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:39.551092 16938 ts_tablet_manager.cc:1398] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.552206 16939 tablet_bootstrap.cc:492] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:39.553023 16939 tablet_bootstrap.cc:654] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.553516 16694 raft_consensus.cc:1050] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:39.553633 16940 tablet_bootstrap.cc:492] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:39.553962 16694 consensus_queue.cc:237] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:39.554473 16940 tablet_bootstrap.cc:654] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.555292 13932 raft_consensus.cc:1261] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.555555 11508 raft_consensus.cc:1261] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.555907 11377 raft_consensus.cc:1261] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.556111 16939 tablet_bootstrap.cc:492] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:39.556186 16939 ts_tablet_manager.cc:1367] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:39.556583 16939 raft_consensus.cc:348] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:39.556718 16939 raft_consensus.cc:374] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.556808 15081 consensus_queue.cc:1035] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.556965 16178 consensus_queue.cc:1035] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.557355 16939 raft_consensus.cc:729] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.557647 16939 consensus_queue.cc:260] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:39.557891 16940 tablet_bootstrap.cc:492] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:39.557873 16694 consensus_queue.cc:1035] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:39.558050 16940 ts_tablet_manager.cc:1367] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:39.558501 16939 ts_tablet_manager.cc:1398] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:39.559677 16940 raft_consensus.cc:348] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:39.559846 16940 raft_consensus.cc:374] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.559916 16940 raft_consensus.cc:729] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.560184 16940 consensus_queue.cc:260] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:39.560534 16950 ts_tablet_manager.cc:902] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:39.561234 16950 tablet_copy_client.cc:308] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.6:46331
I20240501 19:56:39.561599 16940 ts_tablet_manager.cc:1398] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:39.562065 13821 tablet_copy_service.cc:140] P 96fc18dbb5fd41afb9bebf805e8d33a7: Received BeginTabletCopySession request for tablet af9c3a48f92e40f19d75f4151bb4c3e3 from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:48827)
I20240501 19:56:39.562166 13821 tablet_copy_service.cc:161] P 96fc18dbb5fd41afb9bebf805e8d33a7: Beginning new tablet copy session on tablet af9c3a48f92e40f19d75f4151bb4c3e3 from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:48827: session id = 8125d8ad6d0d44caae9c5c90c90df96b-af9c3a48f92e40f19d75f4151bb4c3e3
I20240501 19:56:39.563066 13821 tablet_copy_source_session.cc:205] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.563784 16694 raft_consensus.cc:2932] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.564409 16950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9c3a48f92e40f19d75f4151bb4c3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.566752 11099 catalog_manager.cc:5484] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.566910 11377 raft_consensus.cc:2932] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.567350 16950 tablet_copy_client.cc:783] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.567510 16950 tablet_copy_client.cc:647] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.568722 13931 raft_consensus.cc:2932] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.570402 11643 consensus_queue.cc:237] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:39.571727 11377 raft_consensus.cc:1261] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:39.572130 13931 raft_consensus.cc:1261] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:39.572161 15806 consensus_queue.cc:1035] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.572662 15806 consensus_queue.cc:1035] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:39.574102 16178 raft_consensus.cc:2932] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.576191 16950 tablet_copy_client.cc:523] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.576089 11377 raft_consensus.cc:2932] T d505c415daa54005b8ed22bdc11ce135 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.576243 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d505c415daa54005b8ed22bdc11ce135 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.576541 13931 raft_consensus.cc:2932] T d505c415daa54005b8ed22bdc11ce135 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:39.577842 16950 tablet_bootstrap.cc:492] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:39.578199 11103 catalog_manager.cc:5484] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.580041 16195 raft_consensus.cc:482] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.580137 16195 raft_consensus.cc:504] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:39.580554 16195 leader_election.cc:288] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:39.581545 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546565eab21645c2add5eb6726f9e83d" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:39.581673 13669 raft_consensus.cc:2454] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:39.581713 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet d505c415daa54005b8ed22bdc11ce135 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:39.582166 13996 leader_election.cc:302] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:39.582265 16965 tablet_replica.cc:330] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:39.582331 16195 raft_consensus.cc:2781] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.582384 16195 raft_consensus.cc:482] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.582427 16195 raft_consensus.cc:3037] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.582419 16965 raft_consensus.cc:2229] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.582567 16965 pending_rounds.cc:62] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: Trying to abort 1 pending ops.
I20240501 19:56:39.582613 16965 pending_rounds.cc:68] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022974564571820032 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d505c415daa54005b8ed22bdc11ce135" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } } }
I20240501 19:56:39.582877 16965 raft_consensus.cc:2894] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 19:56:39.582953 16965 raft_consensus.cc:2866] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:39.583016 16965 raft_consensus.cc:2258] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.583317 16195 raft_consensus.cc:504] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:39.583696 16195 leader_election.cc:288] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:39.584012 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546565eab21645c2add5eb6726f9e83d" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:39.584120 13669 raft_consensus.cc:3037] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.584336 16950 tablet_bootstrap.cc:492] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.584542 16950 tablet_bootstrap.cc:492] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:39.584705 16950 ts_tablet_manager.cc:1367] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240501 19:56:39.584972 13669 raft_consensus.cc:2454] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:39.585170 16950 raft_consensus.cc:348] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:39.585247 13996 leader_election.cc:302] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:39.585273 16950 raft_consensus.cc:729] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:39.585407 16195 raft_consensus.cc:2781] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.585379 16950 consensus_queue.cc:260] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:39.585489 16195 raft_consensus.cc:686] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:39.585640 16950 ts_tablet_manager.cc:1398] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:39.585621 16195 consensus_queue.cc:237] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:39.586122 13821 tablet_copy_service.cc:342] P 96fc18dbb5fd41afb9bebf805e8d33a7: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-af9c3a48f92e40f19d75f4151bb4c3e3 received from {username='slave'} at 127.5.40.5:48827
I20240501 19:56:39.586202 13821 tablet_copy_service.cc:433] P 96fc18dbb5fd41afb9bebf805e8d33a7: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-af9c3a48f92e40f19d75f4151bb4c3e3 on tablet af9c3a48f92e40f19d75f4151bb4c3e3 with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:39.586546 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:39.587558 11100 catalog_manager.cc:5484] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.588871 16965 ts_tablet_manager.cc:1875] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.589118 15765 raft_consensus.cc:979] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: : Instructing follower 8ad18cd54c6340fd92b874157875fa49 to start an election
I20240501 19:56:39.589191 15765 raft_consensus.cc:1067] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Signalling peer 8ad18cd54c6340fd92b874157875fa49 to start an election
I20240501 19:56:39.589752 11643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e63472cfff449089d88d18b9e495912"
dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
from {username='slave'} at 127.5.40.3:55181
I20240501 19:56:39.589893 11643 raft_consensus.cc:482] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:39.589951 11643 raft_consensus.cc:3037] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:39.590955 11643 raft_consensus.cc:504] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:39.591377 11643 leader_election.cc:288] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:39.591944 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e63472cfff449089d88d18b9e495912" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:39.592072 14063 raft_consensus.cc:3037] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:39.592114 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e63472cfff449089d88d18b9e495912" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:39.592214 11508 raft_consensus.cc:3032] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:39.592252 11508 raft_consensus.cc:729] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:39.592399 11508 consensus_queue.cc:260] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:39.592609 11508 raft_consensus.cc:3037] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:39.592657 16965 ts_tablet_manager.cc:1888] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:39.592708 16965 log.cc:1196] T d505c415daa54005b8ed22bdc11ce135 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/d505c415daa54005b8ed22bdc11ce135
I20240501 19:56:39.593008 14063 raft_consensus.cc:2454] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 2.
I20240501 19:56:39.593426 11508 raft_consensus.cc:2454] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 2.
I20240501 19:56:39.593845 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet d505c415daa54005b8ed22bdc11ce135 (table rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]) successfully deleted
I20240501 19:56:39.594084 11577 leader_election.cc:302] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 533908e517b24f77b2af8b5863e4ae5a, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:39.594251 16694 raft_consensus.cc:2781] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:39.594326 16694 raft_consensus.cc:686] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 2 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:39.594460 16694 consensus_queue.cc:237] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:39.594727 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546565eab21645c2add5eb6726f9e83d" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:39.594784 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "546565eab21645c2add5eb6726f9e83d" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:39.594848 13931 raft_consensus.cc:2454] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:39.595569 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e63472cfff449089d88d18b9e495912" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:39.595676 13669 raft_consensus.cc:3037] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:39.596515 13669 raft_consensus.cc:2454] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 2.
I20240501 19:56:39.597433 11099 catalog_manager.cc:5484] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 1 to 2, leader changed from 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 2 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.601116 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 33ae450510f54c7baca85259ef97bfa1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=03caa0537ce047f6a42908fc4ead369e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.601402 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ae450510f54c7baca85259ef97bfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.601687 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc4ce342be94b309a35450d3c59a6ce (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=03caa0537ce047f6a42908fc4ead369e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.601755 13911 tablet_service.cc:1467] Processing CreateTablet for tablet 33ae450510f54c7baca85259ef97bfa1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=03caa0537ce047f6a42908fc4ead369e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.601933 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc4ce342be94b309a35450d3c59a6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.601985 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ae450510f54c7baca85259ef97bfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.603520 16940 tablet_bootstrap.cc:492] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:39.604050 16938 tablet_bootstrap.cc:492] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:39.604090 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc4ce342be94b309a35450d3c59a6ce (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=03caa0537ce047f6a42908fc4ead369e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.604351 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc4ce342be94b309a35450d3c59a6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.604434 16940 tablet_bootstrap.cc:654] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.604912 16938 tablet_bootstrap.cc:654] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.605221 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 0da88b92a3fc4f1a89e7eb990557d7d7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=03caa0537ce047f6a42908fc4ead369e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.605492 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da88b92a3fc4f1a89e7eb990557d7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.605604 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 33ae450510f54c7baca85259ef97bfa1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=03caa0537ce047f6a42908fc4ead369e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.605782 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ae450510f54c7baca85259ef97bfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.607218 16970 tablet_bootstrap.cc:492] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:39.607717 16971 tablet_bootstrap.cc:492] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:39.608074 16970 tablet_bootstrap.cc:654] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.608393 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 0da88b92a3fc4f1a89e7eb990557d7d7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=03caa0537ce047f6a42908fc4ead369e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.608644 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da88b92a3fc4f1a89e7eb990557d7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.608650 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc4ce342be94b309a35450d3c59a6ce (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=03caa0537ce047f6a42908fc4ead369e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.608845 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc4ce342be94b309a35450d3c59a6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.609439 14042 tablet_service.cc:1467] Processing CreateTablet for tablet 0da88b92a3fc4f1a89e7eb990557d7d7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=03caa0537ce047f6a42908fc4ead369e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:39.609515 16970 tablet_bootstrap.cc:492] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:39.609586 16970 ts_tablet_manager.cc:1367] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:39.609659 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da88b92a3fc4f1a89e7eb990557d7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.610018 16970 raft_consensus.cc:348] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.610114 16970 raft_consensus.cc:374] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.610139 16970 raft_consensus.cc:729] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.610247 16970 consensus_queue.cc:260] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.610513 16970 ts_tablet_manager.cc:1398] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.611145 16971 tablet_bootstrap.cc:654] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.612344 16938 tablet_bootstrap.cc:492] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:39.612425 16938 ts_tablet_manager.cc:1367] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:39.612835 16938 raft_consensus.cc:348] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:39.612943 16938 raft_consensus.cc:374] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.612990 16938 raft_consensus.cc:729] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.613102 16938 consensus_queue.cc:260] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:39.613382 16938 ts_tablet_manager.cc:1398] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.613539 16938 tablet_bootstrap.cc:492] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:39.614415 16938 tablet_bootstrap.cc:654] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.614542 16940 tablet_bootstrap.cc:492] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:39.614626 16940 ts_tablet_manager.cc:1367] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:39.615041 16940 raft_consensus.cc:348] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:39.615141 16940 raft_consensus.cc:374] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.615170 16940 raft_consensus.cc:729] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.615278 16940 consensus_queue.cc:260] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:39.615602 16940 ts_tablet_manager.cc:1398] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.617806 16938 tablet_bootstrap.cc:492] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:39.617986 16938 ts_tablet_manager.cc:1367] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:39.618391 16938 raft_consensus.cc:348] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.618479 16938 raft_consensus.cc:374] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.618510 16938 raft_consensus.cc:729] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.618619 16938 consensus_queue.cc:260] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.618927 16971 tablet_bootstrap.cc:492] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:39.619020 16971 ts_tablet_manager.cc:1367] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:56:39.619580 16938 ts_tablet_manager.cc:1398] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:39.619688 16971 raft_consensus.cc:348] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.619799 16971 raft_consensus.cc:374] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.619835 16971 raft_consensus.cc:729] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.619951 16971 consensus_queue.cc:260] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.620070 16938 tablet_bootstrap.cc:492] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:39.621264 16971 ts_tablet_manager.cc:1398] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:39.621479 16971 tablet_bootstrap.cc:492] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:39.621678 16938 tablet_bootstrap.cc:654] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.622351 16967 tablet_bootstrap.cc:492] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:39.622378 16971 tablet_bootstrap.cc:654] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.623279 16967 tablet_bootstrap.cc:654] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.624317 16938 tablet_bootstrap.cc:492] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:39.624331 16967 tablet_bootstrap.cc:492] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:39.624401 16938 ts_tablet_manager.cc:1367] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:39.624413 16967 ts_tablet_manager.cc:1367] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:39.624444 16971 tablet_bootstrap.cc:492] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:39.624526 16971 ts_tablet_manager.cc:1367] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:39.624810 16938 raft_consensus.cc:348] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.624914 16938 raft_consensus.cc:374] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.624898 16971 raft_consensus.cc:348] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:39.624944 16938 raft_consensus.cc:729] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.624980 16971 raft_consensus.cc:374] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.625008 16971 raft_consensus.cc:729] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.625066 16938 consensus_queue.cc:260] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.624996 16967 raft_consensus.cc:348] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.625144 16967 raft_consensus.cc:374] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.625118 16971 consensus_queue.cc:260] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:39.625176 16967 raft_consensus.cc:729] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.625330 16938 ts_tablet_manager.cc:1398] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.625300 16967 consensus_queue.cc:260] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.625356 16971 ts_tablet_manager.cc:1398] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.625558 16967 ts_tablet_manager.cc:1398] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.626245 16967 tablet_bootstrap.cc:492] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:39.627141 16967 tablet_bootstrap.cc:654] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.628793 16967 tablet_bootstrap.cc:492] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:39.628872 16967 ts_tablet_manager.cc:1367] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:39.629287 16967 raft_consensus.cc:348] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.629380 16967 raft_consensus.cc:374] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.629411 16967 raft_consensus.cc:729] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.629526 16967 consensus_queue.cc:260] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.629788 16967 ts_tablet_manager.cc:1398] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:39.639633 16296 raft_consensus.cc:979] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: : Instructing follower 533908e517b24f77b2af8b5863e4ae5a to start an election
I20240501 19:56:39.639746 16296 raft_consensus.cc:1067] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Signalling peer 533908e517b24f77b2af8b5863e4ae5a to start an election
I20240501 19:56:39.640266 14063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96511455168148b7b619670f635d3a3e"
dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
from {username='slave'} at 127.5.40.5:52611
I20240501 19:56:39.640434 14063 raft_consensus.cc:482] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:39.640512 14063 raft_consensus.cc:3037] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:39.641602 14063 raft_consensus.cc:504] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:39.642067 14063 leader_election.cc:288] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:39.642436 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96511455168148b7b619670f635d3a3e" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:39.642436 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96511455168148b7b619670f635d3a3e" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:39.642563 11247 raft_consensus.cc:3037] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:39.642562 13669 raft_consensus.cc:3032] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:39.642621 13669 raft_consensus.cc:729] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:39.642750 13669 consensus_queue.cc:260] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:39.642920 13669 raft_consensus.cc:3037] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:39.643743 11247 raft_consensus.cc:2454] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 2.
I20240501 19:56:39.643817 13669 raft_consensus.cc:2454] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 2.
I20240501 19:56:39.644027 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96511455168148b7b619670f635d3a3e" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:39.644169 11643 raft_consensus.cc:3037] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:39.645088 11643 raft_consensus.cc:2454] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 2.
I20240501 19:56:39.645390 13997 leader_election.cc:302] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:39.645555 16567 raft_consensus.cc:2781] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:39.645664 16567 raft_consensus.cc:686] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 2 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:39.645815 16567 consensus_queue.cc:237] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:39.646093 11508 consensus_queue.cc:237] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:39.647629 13669 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:56:39.648044 11103 catalog_manager.cc:5484] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 1 to 2, leader changed from 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 2 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.648195 11643 raft_consensus.cc:1261] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:56:39.648566 15765 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:39.650660 15765 consensus_queue.cc:1035] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:39.651062 13437 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:39.651757 13669 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:39.652765 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 169db26dfa3a4ec991c425c870e669eb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.653769 11103 catalog_manager.cc:5484] T 169db26dfa3a4ec991c425c870e669eb P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 5 to 7, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 2 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.654803 11643 raft_consensus.cc:2932] T 169db26dfa3a4ec991c425c870e669eb P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:39.657330 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 169db26dfa3a4ec991c425c870e669eb with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:39.658018 17004 tablet_replica.cc:330] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:39.658190 17004 raft_consensus.cc:2229] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.658316 17004 raft_consensus.cc:2258] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.659009 17004 ts_tablet_manager.cc:1875] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.661471 17004 ts_tablet_manager.cc:1888] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:39.661605 17004 log.cc:1196] T 169db26dfa3a4ec991c425c870e669eb P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/169db26dfa3a4ec991c425c870e669eb
I20240501 19:56:39.662184 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 169db26dfa3a4ec991c425c870e669eb (table rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]) successfully deleted
I20240501 19:56:39.698314 11247 consensus_queue.cc:237] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:39.699888 13669 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:39.699940 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c07d9fc05254a6399afcc54b3efb8ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:39.700125 11508 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:39.700263 17033 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:39.700529 16188 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:39.702095 17032 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.703938 13669 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.705466 11508 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.706463 11103 catalog_manager.cc:5484] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:39.713362 11099 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:39.718174 13932 raft_consensus.cc:3037] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.719214 13932 raft_consensus.cc:1261] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.719722 16567 consensus_queue.cc:1035] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.720006 13669 raft_consensus.cc:1261] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:39.720458 16567 consensus_queue.cc:1035] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.722702 11099 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet 546565eab21645c2add5eb6726f9e83d (table rebalancer_dynamic_table [id=e1afee203712425fa19eb1ee8ec389ea]). Expected version 1 got 0
I20240501 19:56:39.723215 11100 catalog_manager.cc:5576] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501) does not have the latest schema for tablet 546565eab21645c2add5eb6726f9e83d (table rebalancer_dynamic_table [id=e1afee203712425fa19eb1ee8ec389ea]). Expected version 1 got 0
I20240501 19:56:39.723834 16657 tablet.cc:1722] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:39.724165 16657 tablet.cc:1620] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:39.724377 16333 tablet.cc:1722] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:39.724663 16333 tablet.cc:1620] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:39.729379 11099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=e1afee203712425fa19eb1ee8ec389ea] alter complete (version 1)
I20240501 19:56:39.733314 16334 tablet.cc:1722] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:39.733625 16334 tablet.cc:1620] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:39.735469 16195 raft_consensus.cc:482] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.735560 16195 raft_consensus.cc:504] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.735991 16195 leader_election.cc:288] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:39.736441 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc4ce342be94b309a35450d3c59a6ce" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:39.736585 11377 raft_consensus.cc:2454] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:39.736891 13998 leader_election.cc:302] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:39.737062 16195 raft_consensus.cc:2781] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.737061 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc4ce342be94b309a35450d3c59a6ce" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:39.737114 16195 raft_consensus.cc:482] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.737161 16195 raft_consensus.cc:3037] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.737203 13800 raft_consensus.cc:2454] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:39.738088 16195 raft_consensus.cc:504] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.738389 16195 leader_election.cc:288] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:39.738858 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc4ce342be94b309a35450d3c59a6ce" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:39.738960 11377 raft_consensus.cc:3037] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.739559 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc4ce342be94b309a35450d3c59a6ce" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:39.739677 13800 raft_consensus.cc:3037] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.739915 11377 raft_consensus.cc:2454] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:39.740183 13998 leader_election.cc:302] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:39.740339 16195 raft_consensus.cc:2781] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.740429 16195 raft_consensus.cc:686] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:39.740571 13800 raft_consensus.cc:2454] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:39.740563 16195 consensus_queue.cc:237] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.743937 11103 catalog_manager.cc:5484] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.829438 17114 ts_tablet_manager.cc:902] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:39.829917 17114 tablet_copy_client.cc:308] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:39.830341 11400 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet e6c1d3814e364e718449c3e4379b31ed from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:59475)
I20240501 19:56:39.830446 11400 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet e6c1d3814e364e718449c3e4379b31ed from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:59475: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-e6c1d3814e364e718449c3e4379b31ed
I20240501 19:56:39.831193 11400 tablet_copy_source_session.cc:205] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.832036 17114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c1d3814e364e718449c3e4379b31ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.834004 17114 tablet_copy_client.cc:783] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.834156 17114 tablet_copy_client.cc:647] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.835249 17114 tablet_copy_client.cc:523] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.836199 11103 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:39.836400 17114 tablet_bootstrap.cc:492] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:39.844746 16657 tablet.cc:1722] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:39.845073 16657 tablet.cc:1620] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:39.845561 16333 tablet.cc:1722] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:39.845826 16333 tablet.cc:1620] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:39.848721 16334 tablet.cc:1722] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:39.849002 16334 tablet.cc:1620] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: MemRowSet was empty: no flush needed.
I20240501 19:56:39.849964 11103 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=e1afee203712425fa19eb1ee8ec389ea] alter complete (version 2)
I20240501 19:56:39.858304 16916 ts_tablet_manager.cc:902] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:39.858861 16916 tablet_copy_client.cc:308] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:39.859369 11400 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet 01a10926bce14cf1bd12f21287e08c89 from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:36355)
I20240501 19:56:39.859577 11400 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet 01a10926bce14cf1bd12f21287e08c89 from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:36355: session id = 6921cf2e41f64d66842885512da8de6c-01a10926bce14cf1bd12f21287e08c89
I20240501 19:56:39.860497 11400 tablet_copy_source_session.cc:205] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.861627 16916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a10926bce14cf1bd12f21287e08c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.861732 11643 raft_consensus.cc:1203] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:39.863783 16916 tablet_copy_client.cc:783] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.863916 16916 tablet_copy_client.cc:647] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.867651 16916 tablet_copy_client.cc:523] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.868836 17114 tablet_bootstrap.cc:492] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.868917 16916 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:39.869130 17114 tablet_bootstrap.cc:492] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:39.869328 17114 ts_tablet_manager.cc:1367] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.000s
I20240501 19:56:39.869825 17114 raft_consensus.cc:348] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:39.869954 17114 raft_consensus.cc:729] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:39.870357 17114 consensus_queue.cc:260] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:39.871253 17114 ts_tablet_manager.cc:1398] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:39.871668 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-e6c1d3814e364e718449c3e4379b31ed received from {username='slave'} at 127.5.40.6:59475
I20240501 19:56:39.871755 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-e6c1d3814e364e718449c3e4379b31ed on tablet e6c1d3814e364e718449c3e4379b31ed with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:39.875301 14063 raft_consensus.cc:1203] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:39.880421 11377 raft_consensus.cc:1203] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:39.881043 16694 raft_consensus.cc:1050] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:39.881441 16694 consensus_queue.cc:237] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:39.883013 13800 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:39.883574 11508 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:39.883574 13932 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:39.883770 15081 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:39.884423 15081 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:39.884577 17120 raft_consensus.cc:482] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.884665 17120 raft_consensus.cc:504] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:39.885136 16178 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:39.885285 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ae450510f54c7baca85259ef97bfa1" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:39.885428 13932 raft_consensus.cc:2454] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:39.885761 13996 leader_election.cc:302] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:39.885926 16567 raft_consensus.cc:2781] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.886003 16567 raft_consensus.cc:482] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.886041 16567 raft_consensus.cc:3037] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.886817 16567 raft_consensus.cc:504] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:39.887120 16567 leader_election.cc:288] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:39.887524 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ae450510f54c7baca85259ef97bfa1" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:39.887627 13932 raft_consensus.cc:3037] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.888113 13931 raft_consensus.cc:1203] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:39.888540 13932 raft_consensus.cc:2454] T 33ae450510f54c7baca85259ef97bfa1 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:39.889070 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ae450510f54c7baca85259ef97bfa1" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:39.889117 13996 leader_election.cc:302] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:39.889192 11247 raft_consensus.cc:3037] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.890133 11247 raft_consensus.cc:2454] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:39.891080 16567 raft_consensus.cc:2781] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.891167 16567 raft_consensus.cc:686] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:39.891288 16567 consensus_queue.cc:237] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:39.891861 17120 leader_election.cc:288] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:39.892218 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ae450510f54c7baca85259ef97bfa1" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:39.892319 11247 raft_consensus.cc:2362] T 33ae450510f54c7baca85259ef97bfa1 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1. Re-sending same reply.
I20240501 19:56:39.893119 15806 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.893656 11099 catalog_manager.cc:5484] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.895334 15507 raft_consensus.cc:482] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:39.895421 15507 raft_consensus.cc:504] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.895509 13800 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.895798 15507 leader_election.cc:288] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:39.896519 11508 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.897665 11099 catalog_manager.cc:5484] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.898428 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da88b92a3fc4f1a89e7eb990557d7d7" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:39.898571 14063 raft_consensus.cc:2454] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:39.899183 13734 leader_election.cc:302] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:39.899514 13931 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.899912 16352 raft_consensus.cc:2781] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:39.899961 16352 raft_consensus.cc:482] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:39.900005 16352 raft_consensus.cc:3037] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.900877 16352 raft_consensus.cc:504] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.901214 16352 leader_election.cc:288] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:39.901252 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da88b92a3fc4f1a89e7eb990557d7d7" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:39.901381 11247 raft_consensus.cc:2454] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:39.902706 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da88b92a3fc4f1a89e7eb990557d7d7" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:39.902837 14063 raft_consensus.cc:3037] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.903749 14063 raft_consensus.cc:2454] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:39.904640 13734 leader_election.cc:302] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:39.904824 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da88b92a3fc4f1a89e7eb990557d7d7" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:39.904945 11247 raft_consensus.cc:3037] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.904960 16352 raft_consensus.cc:2781] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:39.905040 16352 raft_consensus.cc:686] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:39.905161 16352 consensus_queue.cc:237] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:39.905743 16916 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:39.905901 11247 raft_consensus.cc:2454] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:39.905980 16916 tablet_bootstrap.cc:492] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:39.906147 16916 ts_tablet_manager.cc:1367] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.000s
I20240501 19:56:39.906574 16916 raft_consensus.cc:348] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:39.906675 16916 raft_consensus.cc:729] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:39.906756 16916 consensus_queue.cc:260] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:39.907166 16916 ts_tablet_manager.cc:1398] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.907759 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-01a10926bce14cf1bd12f21287e08c89 received from {username='slave'} at 127.5.40.7:36355
I20240501 19:56:39.907838 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-01a10926bce14cf1bd12f21287e08c89 on tablet 01a10926bce14cf1bd12f21287e08c89 with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:39.907749 11097 catalog_manager.cc:5484] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.908666 11643 consensus_queue.cc:237] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:39.910183 13931 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:39.910873 13800 raft_consensus.cc:1261] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:39.911099 15806 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:39.911310 16178 consensus_queue.cc:1035] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:39.912933 16178 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.913386 13800 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.914073 13931 raft_consensus.cc:2932] T 69facf86bba24f209c06afafc1ba8f7b P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.914556 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 69facf86bba24f209c06afafc1ba8f7b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.916291 11097 catalog_manager.cc:5484] T 69facf86bba24f209c06afafc1ba8f7b P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:39.918412 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 69facf86bba24f209c06afafc1ba8f7b with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:39.918712 16965 tablet_replica.cc:330] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:39.918828 16965 raft_consensus.cc:2229] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.918967 16965 raft_consensus.cc:2258] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.919647 16965 ts_tablet_manager.cc:1875] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.921233 16965 ts_tablet_manager.cc:1888] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:39.921296 16965 log.cc:1196] T 69facf86bba24f209c06afafc1ba8f7b P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/69facf86bba24f209c06afafc1ba8f7b
I20240501 19:56:39.921747 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 69facf86bba24f209c06afafc1ba8f7b (table rebalance_test_table_1 [id=212d6790a9cd4aecbe3812b6310889c3]) successfully deleted
I20240501 19:56:39.933156 11508 raft_consensus.cc:1203] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:39.956441 11097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:39.962265 16657 tablet.cc:1722] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:39.962649 16334 tablet.cc:1722] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:39.965049 16333 tablet.cc:1722] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:39.967311 11097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=e1afee203712425fa19eb1ee8ec389ea] alter complete (version 3)
I20240501 19:56:39.976838 14063 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:39.977210 16694 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:40.006188 13669 raft_consensus.cc:1203] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:40.014287 17137 ts_tablet_manager.cc:902] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: Initiating tablet copy from peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:40.014745 17137 tablet_copy_client.cc:308] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.5:42501
I20240501 19:56:40.015825 13690 tablet_copy_service.cc:140] P 8125d8ad6d0d44caae9c5c90c90df96b: Received BeginTabletCopySession request for tablet 811d018fde314227a378f2d530e0a924 from peer 10ee60ead7b94c93ab749ec16a04b0dc ({username='slave'} at 127.5.40.1:50477)
I20240501 19:56:40.015929 13690 tablet_copy_service.cc:161] P 8125d8ad6d0d44caae9c5c90c90df96b: Beginning new tablet copy session on tablet 811d018fde314227a378f2d530e0a924 from peer 10ee60ead7b94c93ab749ec16a04b0dc at {username='slave'} at 127.5.40.1:50477: session id = 10ee60ead7b94c93ab749ec16a04b0dc-811d018fde314227a378f2d530e0a924
I20240501 19:56:40.016937 13690 tablet_copy_source_session.cc:205] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:40.017833 17137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811d018fde314227a378f2d530e0a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.021152 17137 tablet_copy_client.cc:783] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:40.021363 17137 tablet_copy_client.cc:647] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:40.024844 17137 tablet_copy_client.cc:523] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:40.026940 17137 tablet_bootstrap.cc:492] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:40.030483 17137 tablet_bootstrap.cc:492] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:40.030730 17137 tablet_bootstrap.cc:492] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap complete.
I20240501 19:56:40.030910 17137 ts_tablet_manager.cc:1367] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:40.031340 17137 raft_consensus.cc:348] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:40.031522 17137 raft_consensus.cc:729] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: LEARNER
I20240501 19:56:40.031642 17137 consensus_queue.cc:260] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:40.031929 17137 ts_tablet_manager.cc:1398] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:40.033221 13690 tablet_copy_service.cc:342] P 8125d8ad6d0d44caae9c5c90c90df96b: Request end of tablet copy session 10ee60ead7b94c93ab749ec16a04b0dc-811d018fde314227a378f2d530e0a924 received from {username='slave'} at 127.5.40.1:50477
I20240501 19:56:40.033300 13690 tablet_copy_service.cc:433] P 8125d8ad6d0d44caae9c5c90c90df96b: ending tablet copy session 10ee60ead7b94c93ab749ec16a04b0dc-811d018fde314227a378f2d530e0a924 on tablet 811d018fde314227a378f2d530e0a924 with peer 10ee60ead7b94c93ab749ec16a04b0dc
I20240501 19:56:40.047329 16950 ts_tablet_manager.cc:902] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:40.047827 16950 tablet_copy_client.cc:308] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:40.048275 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 6f594c53151b4d00a1784b6366e2814a from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:53821)
I20240501 19:56:40.048383 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 6f594c53151b4d00a1784b6366e2814a from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:53821: session id = 8125d8ad6d0d44caae9c5c90c90df96b-6f594c53151b4d00a1784b6366e2814a
I20240501 19:56:40.049355 11533 tablet_copy_source_session.cc:205] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:40.050274 16950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f594c53151b4d00a1784b6366e2814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.052634 16950 tablet_copy_client.cc:783] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:40.052820 16950 tablet_copy_client.cc:647] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:40.054250 16950 tablet_copy_client.cc:523] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:40.055657 16950 tablet_bootstrap.cc:492] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:40.068778 11643 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:40.068806 11097 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:40.068934 11097 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:40.069389 16195 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.071622 11097 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 546565eab21645c2add5eb6726f9e83d
I20240501 19:56:40.073388 13911 tablet_service.cc:1514] Processing DeleteTablet for tablet 546565eab21645c2add5eb6726f9e83d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:40 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:40.073443 14042 tablet_service.cc:1514] Processing DeleteTablet for tablet 546565eab21645c2add5eb6726f9e83d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:40 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:40.073717 13648 tablet_service.cc:1514] Processing DeleteTablet for tablet 546565eab21645c2add5eb6726f9e83d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:40 UTC) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:40.073925 17183 tablet_replica.cc:330] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:40.074065 17183 raft_consensus.cc:2229] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.074123 17184 tablet_replica.cc:330] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:40.074270 17183 raft_consensus.cc:2258] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.074527 17185 tablet_replica.cc:330] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:40.074340 17184 raft_consensus.cc:2229] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:40.074713 17185 raft_consensus.cc:2229] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.074862 17185 raft_consensus.cc:2258] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.075156 17183 ts_tablet_manager.cc:1875] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:40.075420 17185 ts_tablet_manager.cc:1875] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:40.075904 17184 raft_consensus.cc:2258] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.076536 17184 ts_tablet_manager.cc:1875] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:40.077754 17185 ts_tablet_manager.cc:1888] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:40.077876 17185 log.cc:1196] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/546565eab21645c2add5eb6726f9e83d
I20240501 19:56:40.078208 17185 ts_tablet_manager.cc:1909] T 546565eab21645c2add5eb6726f9e83d P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting consensus metadata
I20240501 19:56:40.078944 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet 546565eab21645c2add5eb6726f9e83d (table rebalancer_dynamic_table_new_name [id=e1afee203712425fa19eb1ee8ec389ea]) successfully deleted
I20240501 19:56:40.080204 17183 ts_tablet_manager.cc:1888] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:40.080204 17184 ts_tablet_manager.cc:1888] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:40.080278 17184 log.cc:1196] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/546565eab21645c2add5eb6726f9e83d
I20240501 19:56:40.080415 17183 log.cc:1196] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/546565eab21645c2add5eb6726f9e83d
I20240501 19:56:40.080535 17184 ts_tablet_manager.cc:1909] T 546565eab21645c2add5eb6726f9e83d P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:40.080703 17183 ts_tablet_manager.cc:1909] T 546565eab21645c2add5eb6726f9e83d P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:40.081223 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet 546565eab21645c2add5eb6726f9e83d (table rebalancer_dynamic_table_new_name [id=e1afee203712425fa19eb1ee8ec389ea]) successfully deleted
I20240501 19:56:40.081585 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet 546565eab21645c2add5eb6726f9e83d (table rebalancer_dynamic_table_new_name [id=e1afee203712425fa19eb1ee8ec389ea]) successfully deleted
I20240501 19:56:40.082747 16950 tablet_bootstrap.cc:492] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:40.082966 16950 tablet_bootstrap.cc:492] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:40.083112 16950 ts_tablet_manager.cc:1367] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.003s
I20240501 19:56:40.083592 16950 raft_consensus.cc:348] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:40.083716 16950 raft_consensus.cc:729] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:40.083819 16950 consensus_queue.cc:260] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:40.084126 16950 ts_tablet_manager.cc:1398] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:40.084636 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-6f594c53151b4d00a1784b6366e2814a received from {username='slave'} at 127.5.40.5:53821
I20240501 19:56:40.084717 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-6f594c53151b4d00a1784b6366e2814a on tablet 6f594c53151b4d00a1784b6366e2814a with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:40.093405 11247 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:40.093814 16195 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.098479 11644 raft_consensus.cc:1203] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:40.099633 13669 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:40.100683 14063 consensus_queue.cc:237] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:40.102190 11644 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:40.102560 16567 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.102728 11247 raft_consensus.cc:1261] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:40.103096 16567 consensus_queue.cc:1035] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.104576 16195 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 533908e517b24f77b2af8b5863e4ae5a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.104988 11247 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.107172 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96511455168148b7b619670f635d3a3e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.108336 11097 catalog_manager.cc:5484] T 96511455168148b7b619670f635d3a3e P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 3 to 5, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New cstate: current_term: 2 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.110996 13648 tablet_service.cc:1514] Processing DeleteTablet for tablet 96511455168148b7b619670f635d3a3e with delete_type TABLET_DATA_TOMBSTONED (TS 8125d8ad6d0d44caae9c5c90c90df96b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:40.110921 11644 raft_consensus.cc:2932] T 96511455168148b7b619670f635d3a3e P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.111358 17185 tablet_replica.cc:330] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:40.111532 17185 raft_consensus.cc:2229] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.111646 17185 raft_consensus.cc:2258] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.112309 17185 ts_tablet_manager.cc:1875] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.114279 17185 ts_tablet_manager.cc:1888] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:40.114332 17185 log.cc:1196] T 96511455168148b7b619670f635d3a3e P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/96511455168148b7b619670f635d3a3e
I20240501 19:56:40.114730 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet 96511455168148b7b619670f635d3a3e (table rebalancer_extra_table_2 [id=1a4a8571161f4dc59ff125ba247d2436]) successfully deleted
I20240501 19:56:40.135866 13931 consensus_queue.cc:237] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:40.137316 14063 raft_consensus.cc:1261] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.137379 13800 raft_consensus.cc:1261] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:40.137598 13865 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c -> Peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): Couldn't send request to peer 08711534813d47ba85aaec0771ed5d0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba2b4dda83bc408eaddec5ebc7b734d5. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:40.137933 16340 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:40.138128 16341 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:40.139899 16655 raft_consensus.cc:2932] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:40.140312 14063 raft_consensus.cc:2932] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:40.140513 13800 raft_consensus.cc:2932] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:40.143333 11099 catalog_manager.cc:5484] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:40.144814 13669 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:40.145812 16195 consensus_queue.cc:1035] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.145793 16694 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:40.149350 11508 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:40.149749 16178 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:40.151405 11644 consensus_queue.cc:237] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:40.152515 16195 consensus_queue.cc:1035] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.153086 14063 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:40.153398 13669 raft_consensus.cc:1261] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:40.155706 16178 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:40.155998 16178 consensus_queue.cc:1035] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:40.157840 16178 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.158321 14063 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.159051 13669 raft_consensus.cc:2932] T 5e63472cfff449089d88d18b9e495912 P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.160898 11100 catalog_manager.cc:5484] T 5e63472cfff449089d88d18b9e495912 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 8 to 10, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 2 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.161309 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e63472cfff449089d88d18b9e495912 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.162886 11508 consensus_queue.cc:237] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:40.163813 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e63472cfff449089d88d18b9e495912 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:40.164750 16965 tablet_replica.cc:330] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:40.164762 13931 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:40.164887 16965 raft_consensus.cc:2229] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.164976 16965 raft_consensus.cc:2258] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.165123 15765 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:40.165573 11644 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:40.165622 16965 ts_tablet_manager.cc:1875] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.166155 15765 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:56:40.166433 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00b7dd445e2846d5a0e1277e160d46bc. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:40.167783 16965 ts_tablet_manager.cc:1888] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 19:56:40.167887 16965 log.cc:1196] T 5e63472cfff449089d88d18b9e495912 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/5e63472cfff449089d88d18b9e495912
I20240501 19:56:40.167960 13437 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:40.168447 11644 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:40.169771 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 5e63472cfff449089d88d18b9e495912 (table rebalance_test_table_7 [id=795a20d0a38e4c13992b34c93d82486d]) successfully deleted
I20240501 19:56:40.170336 13931 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:40.170707 11103 catalog_manager.cc:5484] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 6 to 7, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:40.175259 11103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:40.180498 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 2dacd90dfac04a199914638f4a802259 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8d801b18020f4bc4a5c9deb4752973a5]), partition=
I20240501 19:56:40.180600 14042 tablet_service.cc:1467] Processing CreateTablet for tablet 2dacd90dfac04a199914638f4a802259 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8d801b18020f4bc4a5c9deb4752973a5]), partition=
I20240501 19:56:40.180747 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dacd90dfac04a199914638f4a802259. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.180729 13911 tablet_service.cc:1467] Processing CreateTablet for tablet 2dacd90dfac04a199914638f4a802259 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8d801b18020f4bc4a5c9deb4752973a5]), partition=
I20240501 19:56:40.180816 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dacd90dfac04a199914638f4a802259. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.180938 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dacd90dfac04a199914638f4a802259. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.182628 17251 tablet_bootstrap.cc:492] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:40.182899 17252 tablet_bootstrap.cc:492] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:40.183461 17253 tablet_bootstrap.cc:492] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:40.183640 17251 tablet_bootstrap.cc:654] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.183707 17252 tablet_bootstrap.cc:654] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.184345 17253 tablet_bootstrap.cc:654] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.184715 17251 tablet_bootstrap.cc:492] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:40.184795 17251 ts_tablet_manager.cc:1367] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:40.185205 17251 raft_consensus.cc:348] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.185283 17251 raft_consensus.cc:374] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.185302 17251 raft_consensus.cc:729] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.185426 17253 tablet_bootstrap.cc:492] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:40.185405 17251 consensus_queue.cc:260] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.185492 17253 ts_tablet_manager.cc:1367] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:40.185648 17251 ts_tablet_manager.cc:1398] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.185876 17253 raft_consensus.cc:348] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.186003 17253 raft_consensus.cc:374] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.186045 17253 raft_consensus.cc:729] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.186174 17253 consensus_queue.cc:260] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.186461 17253 ts_tablet_manager.cc:1398] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.196301 17252 tablet_bootstrap.cc:492] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:40.196431 17252 ts_tablet_manager.cc:1367] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 19:56:40.196942 17252 raft_consensus.cc:348] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.197073 17252 raft_consensus.cc:374] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.197113 17252 raft_consensus.cc:729] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.197254 17252 consensus_queue.cc:260] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.197536 17252 ts_tablet_manager.cc:1398] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.246901 13800 raft_consensus.cc:1203] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:40.270049 16916 ts_tablet_manager.cc:902] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:40.270560 16916 tablet_copy_client.cc:308] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.1:42965
I20240501 19:56:40.271055 11269 tablet_copy_service.cc:140] P 10ee60ead7b94c93ab749ec16a04b0dc: Received BeginTabletCopySession request for tablet 1c07d9fc05254a6399afcc54b3efb8ac from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:42125)
I20240501 19:56:40.271160 11269 tablet_copy_service.cc:161] P 10ee60ead7b94c93ab749ec16a04b0dc: Beginning new tablet copy session on tablet 1c07d9fc05254a6399afcc54b3efb8ac from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:42125: session id = 6921cf2e41f64d66842885512da8de6c-1c07d9fc05254a6399afcc54b3efb8ac
I20240501 19:56:40.272028 11269 tablet_copy_source_session.cc:205] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:40.272866 16916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c07d9fc05254a6399afcc54b3efb8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.274847 16916 tablet_copy_client.cc:783] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:40.275012 16916 tablet_copy_client.cc:647] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:40.276528 16916 tablet_copy_client.cc:523] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:40.277102 13931 raft_consensus.cc:1203] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:40.280134 16916 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:40.288062 11511 consensus_queue.cc:237] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
W20240501 19:56:40.289525 11445 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3b35dab7eec4b73b208f687dad46ba2. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:40.289602 11247 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:40.289947 13800 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:40.290309 13437 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:40.290474 15765 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:40.293026 16302 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:40.293404 11247 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:40.296530 11100 catalog_manager.cc:5484] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:40.300259 13800 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:40.309595 16916 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:40.309883 16916 tablet_bootstrap.cc:492] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:40.310074 16916 ts_tablet_manager.cc:1367] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20240501 19:56:40.310580 16916 raft_consensus.cc:348] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:40.310704 16916 raft_consensus.cc:729] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:40.310847 16916 consensus_queue.cc:260] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:40.311844 16916 ts_tablet_manager.cc:1398] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:40.312295 11269 tablet_copy_service.cc:342] P 10ee60ead7b94c93ab749ec16a04b0dc: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-1c07d9fc05254a6399afcc54b3efb8ac received from {username='slave'} at 127.5.40.7:42125
I20240501 19:56:40.312379 11269 tablet_copy_service.cc:433] P 10ee60ead7b94c93ab749ec16a04b0dc: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-1c07d9fc05254a6399afcc54b3efb8ac on tablet 1c07d9fc05254a6399afcc54b3efb8ac with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:40.317951 16195 raft_consensus.cc:482] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.318068 16195 raft_consensus.cc:504] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.318454 16195 leader_election.cc:288] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:40.318753 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dacd90dfac04a199914638f4a802259" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:40.318878 11511 raft_consensus.cc:2454] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:40.318856 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dacd90dfac04a199914638f4a802259" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:40.318969 13932 raft_consensus.cc:2454] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:40.319134 13996 leader_election.cc:302] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 533908e517b24f77b2af8b5863e4ae5a; no voters:
I20240501 19:56:40.319427 16195 raft_consensus.cc:2781] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.319519 16195 raft_consensus.cc:482] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.319554 16195 raft_consensus.cc:3037] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.320546 16195 raft_consensus.cc:504] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.320909 16195 leader_election.cc:288] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:40.321262 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dacd90dfac04a199914638f4a802259" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:40.321272 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dacd90dfac04a199914638f4a802259" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:40.321381 13932 raft_consensus.cc:3037] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.321388 11511 raft_consensus.cc:3037] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.322353 13932 raft_consensus.cc:2454] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:40.322353 11511 raft_consensus.cc:2454] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:40.322619 13996 leader_election.cc:302] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:40.322916 16195 raft_consensus.cc:2781] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.322997 16195 raft_consensus.cc:686] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:40.323122 16195 consensus_queue.cc:237] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.325412 11103 catalog_manager.cc:5484] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:40.431593 16352 raft_consensus.cc:1050] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7: attempting to promote NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 to VOTER
I20240501 19:56:40.431994 16352 consensus_queue.cc:237] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:40.433257 11377 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.433720 17418 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.433859 11247 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.433840 13669 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.434309 17418 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.434487 14794 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.436278 14794 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:40.436758 11247 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:40.438160 13669 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:40.438369 11377 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:40.439553 11103 catalog_manager.cc:5484] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 5 to 6, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.441108 11097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:40.443648 13437 raft_consensus.cc:1050] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 533908e517b24f77b2af8b5863e4ae5a to VOTER
I20240501 19:56:40.443909 13799 consensus_queue.cc:237] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:40.444054 13437 consensus_queue.cc:237] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:40.445096 11247 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:40.445695 11644 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.445884 15507 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:40.446763 13932 raft_consensus.cc:1261] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.446775 11377 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.447250 13155 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.447664 11511 raft_consensus.cc:1261] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.448746 15507 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:40.449131 15765 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.449247 11247 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:40.450104 11377 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:40.450111 14063 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.450407 11092 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8078e0cbffa943c98584224d05b0ece0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.450500 16195 consensus_queue.cc:1035] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.450762 16195 consensus_queue.cc:1035] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.450845 15765 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.453809 11106 catalog_manager.cc:5484] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 6 to 7, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:40.455302 16352 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:40.455713 11097 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet 2dacd90dfac04a199914638f4a802259 (table rebalancer_dynamic_table [id=8d801b18020f4bc4a5c9deb4752973a5]). Expected version 1 got 0
I20240501 19:56:40.456017 11099 catalog_manager.cc:5576] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965) does not have the latest schema for tablet 2dacd90dfac04a199914638f4a802259 (table rebalancer_dynamic_table [id=8d801b18020f4bc4a5c9deb4752973a5]). Expected version 1 got 0
I20240501 19:56:40.456620 13669 raft_consensus.cc:1203] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:40.456912 11380 raft_consensus.cc:2932] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:40.457249 16657 tablet.cc:1722] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:40.457549 16657 tablet.cc:1620] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:40.458140 16333 tablet.cc:1722] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:40.458442 16333 tablet.cc:1620] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:40.461158 17428 tablet.cc:1722] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:40.461546 17428 tablet.cc:1620] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:40.464198 13437 raft_consensus.cc:1050] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:40.464558 13437 consensus_queue.cc:237] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:40.464831 15507 consensus_queue.cc:1035] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.466214 13648 tablet_service.cc:1514] Processing DeleteTablet for tablet 8078e0cbffa943c98584224d05b0ece0 with delete_type TABLET_DATA_TOMBSTONED (TS 8125d8ad6d0d44caae9c5c90c90df96b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35220
I20240501 19:56:40.466470 17185 tablet_replica.cc:330] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b: stopping tablet replica
I20240501 19:56:40.466619 17185 raft_consensus.cc:2229] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.466718 13932 raft_consensus.cc:1261] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.466789 17185 raft_consensus.cc:2258] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.467101 16302 consensus_queue.cc:1035] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.467692 17185 ts_tablet_manager.cc:1875] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.466845 11099 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=8d801b18020f4bc4a5c9deb4752973a5] alter complete (version 1)
I20240501 19:56:40.469189 11380 raft_consensus.cc:1261] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.469712 11247 raft_consensus.cc:1261] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.469749 17185 ts_tablet_manager.cc:1888] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:40.469759 16302 consensus_queue.cc:1035] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.469816 17185 log.cc:1196] T 8078e0cbffa943c98584224d05b0ece0 P 8125d8ad6d0d44caae9c5c90c90df96b: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-4/wal/wals/8078e0cbffa943c98584224d05b0ece0
I20240501 19:56:40.470086 16302 consensus_queue.cc:1035] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.470494 11089 catalog_manager.cc:4830] TS 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): tablet 8078e0cbffa943c98584224d05b0ece0 (table rebalancer_extra_table_4 [id=7da0a470b52842538cd2dbf56382ebd8]) successfully deleted
I20240501 19:56:40.470844 13155 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.473445 15765 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.473948 11377 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.474023 11380 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.476028 11245 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.476078 11644 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.476729 13932 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.478624 16352 consensus_queue.cc:1035] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.479537 11099 catalog_manager.cc:5484] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 4 to 5, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.481614 14063 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.482271 11101 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:40.484159 16352 raft_consensus.cc:1050] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7: attempting to promote NON_VOTER 8ad18cd54c6340fd92b874157875fa49 to VOTER
I20240501 19:56:40.484302 11103 catalog_manager.cc:5484] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.484573 16352 consensus_queue.cc:237] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:40.485988 11644 raft_consensus.cc:1261] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.486536 14794 consensus_queue.cc:1035] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.487042 14063 raft_consensus.cc:1261] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.487558 14794 consensus_queue.cc:1035] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.489503 11511 consensus_queue.cc:237] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } }
I20240501 19:56:40.490603 11644 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.490682 14062 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.491036 13155 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.491205 16302 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.492203 13932 raft_consensus.cc:1261] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.492774 17418 consensus_queue.cc:1035] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.493206 16302 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.493476 14042 tablet_service.cc:1467] Processing CreateTablet for tablet 313fb8dab2644b339cdd22647a8550fc (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6c88a15f03b9492c8c65cb0d0c3387ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.493685 13911 tablet_service.cc:1467] Processing CreateTablet for tablet 33be1f7276c74d49b42c8761e3e4cd4e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6c88a15f03b9492c8c65cb0d0c3387ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.493734 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313fb8dab2644b339cdd22647a8550fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.493724 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 313fb8dab2644b339cdd22647a8550fc (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6c88a15f03b9492c8c65cb0d0c3387ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.493923 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33be1f7276c74d49b42c8761e3e4cd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.493975 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313fb8dab2644b339cdd22647a8550fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.495050 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 33be1f7276c74d49b42c8761e3e4cd4e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6c88a15f03b9492c8c65cb0d0c3387ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.495278 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33be1f7276c74d49b42c8761e3e4cd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.496102 17253 tablet_bootstrap.cc:492] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:40.496163 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 313fb8dab2644b339cdd22647a8550fc (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6c88a15f03b9492c8c65cb0d0c3387ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.496510 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313fb8dab2644b339cdd22647a8550fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.497049 17253 tablet_bootstrap.cc:654] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.497017 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 33be1f7276c74d49b42c8761e3e4cd4e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6c88a15f03b9492c8c65cb0d0c3387ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.497257 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33be1f7276c74d49b42c8761e3e4cd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.497519 17252 tablet_bootstrap.cc:492] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:40.498386 17252 tablet_bootstrap.cc:654] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.498569 17251 tablet_bootstrap.cc:492] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:40.499481 17251 tablet_bootstrap.cc:654] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.499712 16567 consensus_queue.cc:1035] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.499941 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fc7da07b6c5414bb305264962d0f218 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.500510 11644 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.501684 13910 tablet_service.cc:1467] Processing CreateTablet for tablet ee5178fb00d741d49d42168a71e5d23a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6c88a15f03b9492c8c65cb0d0c3387ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.501920 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5178fb00d741d49d42168a71e5d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.502205 13648 tablet_service.cc:1467] Processing CreateTablet for tablet ee5178fb00d741d49d42168a71e5d23a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6c88a15f03b9492c8c65cb0d0c3387ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.502471 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5178fb00d741d49d42168a71e5d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.502853 14042 tablet_service.cc:1467] Processing CreateTablet for tablet ee5178fb00d741d49d42168a71e5d23a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6c88a15f03b9492c8c65cb0d0c3387ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.503096 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5178fb00d741d49d42168a71e5d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.503221 17252 tablet_bootstrap.cc:492] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:40.503307 17252 ts_tablet_manager.cc:1367] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:40.503805 17252 raft_consensus.cc:348] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:40.503902 17252 raft_consensus.cc:374] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.503928 17252 raft_consensus.cc:729] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.504050 17252 consensus_queue.cc:260] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:40.504317 17252 ts_tablet_manager.cc:1398] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:40.504865 17253 tablet_bootstrap.cc:492] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:40.504945 17253 ts_tablet_manager.cc:1367] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:40.505398 17253 raft_consensus.cc:348] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:40.505506 17253 raft_consensus.cc:374] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.505540 17253 raft_consensus.cc:729] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.505652 17253 consensus_queue.cc:260] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:40.505688 14062 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.505929 17253 ts_tablet_manager.cc:1398] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.506088 17253 tablet_bootstrap.cc:492] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:40.506168 16195 consensus_queue.cc:1035] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.506958 17253 tablet_bootstrap.cc:654] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.506959 11105 catalog_manager.cc:5484] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } }
I20240501 19:56:40.508149 17444 tablet_bootstrap.cc:492] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:40.508924 16195 raft_consensus.cc:1050] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a: attempting to promote NON_VOTER 8ad18cd54c6340fd92b874157875fa49 to VOTER
I20240501 19:56:40.509379 16195 consensus_queue.cc:237] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:40.509760 11511 consensus_queue.cc:237] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:40.510890 13930 raft_consensus.cc:1261] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.511315 11245 raft_consensus.cc:1261] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.511363 11644 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.511416 13799 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.511190 17444 tablet_bootstrap.cc:654] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.511629 11249 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.511822 17120 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.511895 15765 consensus_queue.cc:1035] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.512475 17120 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.512632 15765 consensus_queue.cc:1035] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.514159 16567 raft_consensus.cc:1050] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a: attempting to promote NON_VOTER 08711534813d47ba85aaec0771ed5d0f to VOTER
I20240501 19:56:40.514513 16567 consensus_queue.cc:237] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:40.514950 17251 tablet_bootstrap.cc:492] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:40.515035 17251 ts_tablet_manager.cc:1367] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.001s
I20240501 19:56:40.515269 15765 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.515638 17251 raft_consensus.cc:348] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:40.515729 17251 raft_consensus.cc:374] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.515761 17251 raft_consensus.cc:729] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.515621 13930 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.515861 17251 consensus_queue.cc:260] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:40.516062 13800 raft_consensus.cc:1261] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.516155 17251 ts_tablet_manager.cc:1398] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:40.516317 17251 tablet_bootstrap.cc:492] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:40.516481 17120 consensus_queue.cc:1035] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.517292 17251 tablet_bootstrap.cc:654] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.517339 11249 raft_consensus.cc:2932] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.518738 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eff09e5884f44e18a66583e8e0e73679 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.519556 13669 raft_consensus.cc:1261] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.520141 16567 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.520551 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc7da07b6c5414bb305264962d0f218 with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:40.520831 11511 raft_consensus.cc:1261] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.522176 11105 catalog_manager.cc:5484] T eff09e5884f44e18a66583e8e0e73679 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.522455 17120 consensus_queue.cc:1035] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.522791 17120 consensus_queue.cc:1035] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.523387 17251 tablet_bootstrap.cc:492] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:40.523505 17251 ts_tablet_manager.cc:1367] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 19:56:40.523523 17457 tablet_replica.cc:330] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:40.523679 17457 raft_consensus.cc:2229] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.523850 17457 raft_consensus.cc:2258] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.523955 17251 raft_consensus.cc:348] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:40.524044 17251 raft_consensus.cc:374] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.524075 17251 raft_consensus.cc:729] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.524178 17251 consensus_queue.cc:260] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:40.524446 17251 ts_tablet_manager.cc:1398] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.524518 17253 tablet_bootstrap.cc:492] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:40.524600 17253 ts_tablet_manager.cc:1367] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20240501 19:56:40.525040 17253 raft_consensus.cc:348] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:40.525143 17253 raft_consensus.cc:374] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.525177 17253 raft_consensus.cc:729] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.525286 17253 consensus_queue.cc:260] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:40.525589 17253 ts_tablet_manager.cc:1398] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:40.525758 17253 tablet_bootstrap.cc:492] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:40.526450 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet eff09e5884f44e18a66583e8e0e73679 with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:40.526758 17253 tablet_bootstrap.cc:654] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.527241 17444 tablet_bootstrap.cc:492] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:40.527415 17444 ts_tablet_manager.cc:1367] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 19:56:40.528170 17418 raft_consensus.cc:2932] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.529995 17253 tablet_bootstrap.cc:492] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:40.530087 17253 ts_tablet_manager.cc:1367] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:40.530493 17253 raft_consensus.cc:348] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:40.530489 17444 raft_consensus.cc:348] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:40.530587 17253 raft_consensus.cc:374] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.530594 17444 raft_consensus.cc:374] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.530618 17253 raft_consensus.cc:729] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.530623 17444 raft_consensus.cc:729] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.530732 17253 consensus_queue.cc:260] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:40.530988 17253 ts_tablet_manager.cc:1398] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:40.531566 11644 raft_consensus.cc:2932] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.532358 13932 raft_consensus.cc:2932] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.533993 17457 ts_tablet_manager.cc:1875] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.534948 11101 catalog_manager.cc:5484] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.537314 17438 tablet_bootstrap.cc:492] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:40.538043 13800 consensus_queue.cc:237] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:40.538969 17457 ts_tablet_manager.cc:1888] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:40.539036 17457 log.cc:1196] T 1fc7da07b6c5414bb305264962d0f218 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/1fc7da07b6c5414bb305264962d0f218
I20240501 19:56:40.539386 13932 raft_consensus.cc:1261] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:40.539494 11644 raft_consensus.cc:1261] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:40.539584 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 1fc7da07b6c5414bb305264962d0f218 (table rebalance_test_table_9 [id=9b05408d49ba4ecfa29343710c343475]) successfully deleted
I20240501 19:56:40.539762 16352 consensus_queue.cc:1035] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.540076 17457 tablet_replica.cc:330] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:40.540324 16352 consensus_queue.cc:1035] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.540872 17457 raft_consensus.cc:2229] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.541294 17457 raft_consensus.cc:2258] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.542107 17457 ts_tablet_manager.cc:1875] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.542167 16352 raft_consensus.cc:2932] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.530761 17444 consensus_queue.cc:260] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:40.543869 17444 ts_tablet_manager.cc:1398] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240501 19:56:40.544728 11092 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c0ec896817246b19f753f94ff3f7af5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.544682 11644 raft_consensus.cc:2932] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.545491 17438 tablet_bootstrap.cc:654] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.546168 17441 tablet_bootstrap.cc:492] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:40.547295 17441 tablet_bootstrap.cc:654] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.547226 13932 raft_consensus.cc:2932] T 7c0ec896817246b19f753f94ff3f7af5 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.547390 11100 catalog_manager.cc:5484] T 7c0ec896817246b19f753f94ff3f7af5 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 3 to 4, VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) evicted. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.549453 16195 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.550833 16195 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:40.553016 14042 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c0ec896817246b19f753f94ff3f7af5 with delete_type TABLET_DATA_TOMBSTONED (TS 533908e517b24f77b2af8b5863e4ae5a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:40.553215 17184 tablet_replica.cc:330] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:40.553357 17184 raft_consensus.cc:2229] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.553455 17184 pending_rounds.cc:62] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: Trying to abort 1 pending ops.
I20240501 19:56:40.553516 17184 pending_rounds.cc:68] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022974568383655936 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7c0ec896817246b19f753f94ff3f7af5" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } } }
I20240501 19:56:40.553686 17184 raft_consensus.cc:2894] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 19:56:40.553741 17184 raft_consensus.cc:2866] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:40.553783 17184 raft_consensus.cc:2258] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.553651 13669 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:40.553674 11101 catalog_manager.cc:5484] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.554270 17438 tablet_bootstrap.cc:492] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:40.554359 17438 ts_tablet_manager.cc:1367] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:56:40.554311 11644 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.554626 17457 ts_tablet_manager.cc:1888] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:40.554701 17457 log.cc:1196] T eff09e5884f44e18a66583e8e0e73679 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/eff09e5884f44e18a66583e8e0e73679
I20240501 19:56:40.554834 17438 raft_consensus.cc:348] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:40.554929 17438 raft_consensus.cc:374] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.554957 17438 raft_consensus.cc:729] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.555071 17438 consensus_queue.cc:260] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:40.555394 17438 ts_tablet_manager.cc:1398] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:40.555629 11247 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.555799 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet eff09e5884f44e18a66583e8e0e73679 (table rebalance_test_table_13 [id=eac787e40f7a432991b46c06b80e7e26]) successfully deleted
I20240501 19:56:40.555691 11511 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:40.557529 11105 catalog_manager.cc:5484] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:40.558526 13800 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.563634 13800 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:40.566877 14062 consensus_queue.cc:237] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:40.568078 11644 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:40.568249 14062 consensus_queue.cc:237] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:40.569438 16567 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.569813 11511 raft_consensus.cc:1261] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:40.570606 16567 consensus_queue.cc:1035] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.571398 13669 raft_consensus.cc:1261] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:40.571907 16567 consensus_queue.cc:1035] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.572872 13800 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:40.573248 17453 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.574101 17453 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:40.574869 13669 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:40.575075 16195 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.576750 11644 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.577373 11100 catalog_manager.cc:5484] T a210b79583db455d86f2c68ee49aa4b7 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:40.577823 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a210b79583db455d86f2c68ee49aa4b7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.577993 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66bdddd3550b4ae8a97535e4747a5234 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.578277 13800 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:40.579761 11105 catalog_manager.cc:5484] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 3 to 4, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.580108 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet a210b79583db455d86f2c68ee49aa4b7 with delete_type TABLET_DATA_TOMBSTONED (TS 96fc18dbb5fd41afb9bebf805e8d33a7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:40.583588 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bdddd3550b4ae8a97535e4747a5234 with delete_type TABLET_DATA_TOMBSTONED (TS 10ee60ead7b94c93ab749ec16a04b0dc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:40.583706 11511 raft_consensus.cc:2932] T a210b79583db455d86f2c68ee49aa4b7 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:40.585063 17466 tablet_replica.cc:330] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:40.585187 17466 raft_consensus.cc:2229] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.585301 17466 raft_consensus.cc:2258] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.587106 17466 ts_tablet_manager.cc:1875] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.587193 11100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240501 19:56:40.587761 17437 consensus_queue.cc:1159] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "533908e517b24f77b2af8b5863e4ae5a" status { last_received { term: 1 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 3 } }
I20240501 19:56:40.588497 17184 ts_tablet_manager.cc:1875] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.590423 17184 ts_tablet_manager.cc:1888] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:40.590513 17184 log.cc:1196] T 7c0ec896817246b19f753f94ff3f7af5 P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/7c0ec896817246b19f753f94ff3f7af5
I20240501 19:56:40.591023 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet 7c0ec896817246b19f753f94ff3f7af5 (table rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]) successfully deleted
I20240501 19:56:40.592025 17441 tablet_bootstrap.cc:492] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:40.592146 17441 ts_tablet_manager.cc:1367] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.001s
I20240501 19:56:40.592125 17466 ts_tablet_manager.cc:1888] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:40.592202 17466 log.cc:1196] T 66bdddd3550b4ae8a97535e4747a5234 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/66bdddd3550b4ae8a97535e4747a5234
I20240501 19:56:40.592631 17441 raft_consensus.cc:348] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:40.592751 17441 raft_consensus.cc:374] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.593155 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet 66bdddd3550b4ae8a97535e4747a5234 (table rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]) successfully deleted
I20240501 19:56:40.593297 17441 raft_consensus.cc:729] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.592964 17465 tablet_replica.cc:330] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:40.593888 17465 raft_consensus.cc:2229] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.594087 17441 consensus_queue.cc:260] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:40.594292 17465 raft_consensus.cc:2258] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.595320 17465 ts_tablet_manager.cc:1875] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.595999 17441 ts_tablet_manager.cc:1398] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:40.596762 16657 tablet.cc:1722] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:40.597115 16657 tablet.cc:1620] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:40.597275 16333 tablet.cc:1722] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:40.597564 16333 tablet.cc:1620] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:40.597589 17465 ts_tablet_manager.cc:1888] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:40.597668 17465 log.cc:1196] T a210b79583db455d86f2c68ee49aa4b7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/a210b79583db455d86f2c68ee49aa4b7
I20240501 19:56:40.598459 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet a210b79583db455d86f2c68ee49aa4b7 (table rebalancer_extra_table_6 [id=cb33f6f06e0049a4b3859ed29180775f]) successfully deleted
I20240501 19:56:40.598526 11247 raft_consensus.cc:1203] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:40.598917 17428 tablet.cc:1722] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:40.599193 17428 tablet.cc:1620] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:40.605813 11105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=8d801b18020f4bc4a5c9deb4752973a5] alter complete (version 2)
I20240501 19:56:40.606114 17473 ts_tablet_manager.cc:902] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: Initiating tablet copy from peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:40.606789 17473 tablet_copy_client.cc:308] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.7:36243
I20240501 19:56:40.607785 13952 tablet_copy_service.cc:140] P 6921cf2e41f64d66842885512da8de6c: Received BeginTabletCopySession request for tablet ba2b4dda83bc408eaddec5ebc7b734d5 from peer 08711534813d47ba85aaec0771ed5d0f ({username='slave'} at 127.5.40.3:55041)
I20240501 19:56:40.607926 13952 tablet_copy_service.cc:161] P 6921cf2e41f64d66842885512da8de6c: Beginning new tablet copy session on tablet ba2b4dda83bc408eaddec5ebc7b734d5 from peer 08711534813d47ba85aaec0771ed5d0f at {username='slave'} at 127.5.40.3:55041: session id = 08711534813d47ba85aaec0771ed5d0f-ba2b4dda83bc408eaddec5ebc7b734d5
I20240501 19:56:40.608420 15507 raft_consensus.cc:1050] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:40.608845 15507 consensus_queue.cc:237] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:40.609321 13952 tablet_copy_source_session.cc:205] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:40.610635 13669 raft_consensus.cc:1261] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.611131 17473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b4dda83bc408eaddec5ebc7b734d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.611253 11247 raft_consensus.cc:1261] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.611642 14063 raft_consensus.cc:1261] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:40.612066 17418 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.612334 16352 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.612411 17472 raft_consensus.cc:482] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.612525 17472 raft_consensus.cc:504] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:40.612594 15507 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:40.612888 17472 leader_election.cc:288] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:40.613670 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5178fb00d741d49d42168a71e5d23a" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:40.613796 14062 raft_consensus.cc:2454] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:40.614044 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5178fb00d741d49d42168a71e5d23a" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:40.614172 13932 raft_consensus.cc:2454] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:40.614786 17473 tablet_copy_client.cc:783] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:40.614949 17473 tablet_copy_client.cc:647] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:40.615114 13603 leader_election.cc:302] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:40.615298 17472 raft_consensus.cc:2781] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.615360 17472 raft_consensus.cc:482] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.615393 17472 raft_consensus.cc:3037] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.616504 17472 raft_consensus.cc:504] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:40.616847 17472 leader_election.cc:288] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:40.617707 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5178fb00d741d49d42168a71e5d23a" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:40.617841 14062 raft_consensus.cc:3037] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.618716 14062 raft_consensus.cc:2454] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:40.618948 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5178fb00d741d49d42168a71e5d23a" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:40.619066 13932 raft_consensus.cc:3037] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.619105 13603 leader_election.cc:302] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:40.618932 16352 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.619382 14062 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.619565 13669 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.620239 13932 raft_consensus.cc:2454] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:40.620736 13155 raft_consensus.cc:482] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.620820 13155 raft_consensus.cc:504] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:40.621187 13155 leader_election.cc:288] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:40.621577 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33be1f7276c74d49b42c8761e3e4cd4e" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:40.621589 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33be1f7276c74d49b42c8761e3e4cd4e" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:40.621685 13932 raft_consensus.cc:2454] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:40.621685 13800 raft_consensus.cc:2454] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 0.
I20240501 19:56:40.621904 15931 raft_consensus.cc:2781] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.621953 11445 leader_election.cc:302] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:40.622052 15931 raft_consensus.cc:686] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:40.622195 15931 consensus_queue.cc:237] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:40.622524 11249 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.622705 11100 catalog_manager.cc:5484] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 2 to 3, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.623854 13155 raft_consensus.cc:2781] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.623914 13155 raft_consensus.cc:482] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.623952 13155 raft_consensus.cc:3037] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.624817 13155 raft_consensus.cc:504] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:40.625169 13155 leader_election.cc:288] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:40.625463 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33be1f7276c74d49b42c8761e3e4cd4e" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:40.625569 13800 raft_consensus.cc:3037] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.626000 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33be1f7276c74d49b42c8761e3e4cd4e" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:40.626101 13932 raft_consensus.cc:3037] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.626479 13800 raft_consensus.cc:2454] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:40.626756 11448 leader_election.cc:302] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:40.626914 13155 raft_consensus.cc:2781] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.627012 13155 raft_consensus.cc:686] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:40.627027 17473 tablet_copy_client.cc:523] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:40.627039 13932 raft_consensus.cc:2454] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
I20240501 19:56:40.627141 13155 consensus_queue.cc:237] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:40.628264 11106 catalog_manager.cc:5484] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.628468 17473 tablet_bootstrap.cc:492] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:40.629786 13800 consensus_queue.cc:237] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:40.630569 11100 catalog_manager.cc:5484] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.633821 14062 raft_consensus.cc:1261] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:40.634279 13669 raft_consensus.cc:1261] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:40.634675 16352 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.634869 15507 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:40.635366 17473 tablet_bootstrap.cc:492] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:40.635816 17473 tablet_bootstrap.cc:492] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap complete.
I20240501 19:56:40.635987 17473 ts_tablet_manager.cc:1367] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 19:56:40.636472 17473 raft_consensus.cc:348] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:40.636445 15507 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.636600 17473 raft_consensus.cc:729] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: LEARNER
I20240501 19:56:40.636806 17473 consensus_queue.cc:260] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:40.636972 14062 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.637622 17473 ts_tablet_manager.cc:1398] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:40.638110 11092 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af9c3a48f92e40f19d75f4151bb4c3e3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.638163 13952 tablet_copy_service.cc:342] P 6921cf2e41f64d66842885512da8de6c: Request end of tablet copy session 08711534813d47ba85aaec0771ed5d0f-ba2b4dda83bc408eaddec5ebc7b734d5 received from {username='slave'} at 127.5.40.3:55041
I20240501 19:56:40.638253 13952 tablet_copy_service.cc:433] P 6921cf2e41f64d66842885512da8de6c: ending tablet copy session 08711534813d47ba85aaec0771ed5d0f-ba2b4dda83bc408eaddec5ebc7b734d5 on tablet ba2b4dda83bc408eaddec5ebc7b734d5 with peer 08711534813d47ba85aaec0771ed5d0f
I20240501 19:56:40.638672 13669 raft_consensus.cc:2932] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.641276 11101 catalog_manager.cc:5484] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 3 to 4, VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) evicted. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.644237 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet af9c3a48f92e40f19d75f4151bb4c3e3 with delete_type TABLET_DATA_TOMBSTONED (TS 10ee60ead7b94c93ab749ec16a04b0dc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38884
I20240501 19:56:40.644558 17466 tablet_replica.cc:330] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc: stopping tablet replica
I20240501 19:56:40.644702 17466 raft_consensus.cc:2229] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.644840 17466 raft_consensus.cc:2258] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.645457 17466 ts_tablet_manager.cc:1875] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.647651 17466 ts_tablet_manager.cc:1888] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:40.647719 17466 log.cc:1196] T af9c3a48f92e40f19d75f4151bb4c3e3 P 10ee60ead7b94c93ab749ec16a04b0dc: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-0/wal/wals/af9c3a48f92e40f19d75f4151bb4c3e3
I20240501 19:56:40.648249 11089 catalog_manager.cc:4830] TS 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): tablet af9c3a48f92e40f19d75f4151bb4c3e3 (table rebalancer_extra_table_8 [id=8ce2036f169e4eb894a584ac7a036047]) successfully deleted
I20240501 19:56:40.649369 16340 raft_consensus.cc:482] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.649475 16340 raft_consensus.cc:504] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:40.650219 16340 leader_election.cc:288] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:40.650739 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313fb8dab2644b339cdd22647a8550fc" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:40.650892 11644 raft_consensus.cc:2454] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:40.651172 13866 leader_election.cc:302] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:40.651360 16340 raft_consensus.cc:2781] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.651414 16340 raft_consensus.cc:482] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.651487 16340 raft_consensus.cc:3037] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.651736 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313fb8dab2644b339cdd22647a8550fc" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:40.651873 14062 raft_consensus.cc:2454] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:40.652488 16340 raft_consensus.cc:504] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:40.652617 17242 raft_consensus.cc:1050] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:40.652837 16340 leader_election.cc:288] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:40.653059 17242 consensus_queue.cc:237] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:40.653115 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313fb8dab2644b339cdd22647a8550fc" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:40.653214 14062 raft_consensus.cc:3037] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.654119 14062 raft_consensus.cc:2454] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:40.654534 11511 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.655011 16220 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.655236 13800 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.656335 16220 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.657992 11643 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.658334 16220 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:40.658659 13800 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:40.658921 13865 leader_election.cc:302] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:40.659102 16340 raft_consensus.cc:2781] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.659178 16340 raft_consensus.cc:686] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Becoming Leader. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Running, Role: LEADER
I20240501 19:56:40.659168 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313fb8dab2644b339cdd22647a8550fc" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:40.659268 11644 raft_consensus.cc:3037] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.659299 16340 consensus_queue.cc:237] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:40.659590 16220 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.660295 11644 raft_consensus.cc:2454] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:40.661581 11100 catalog_manager.cc:5484] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.661638 11106 catalog_manager.cc:5484] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921cf2e41f64d66842885512da8de6c (127.5.40.7). New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:40.662700 11644 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:40.662693 11511 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:40.667230 11377 consensus_queue.cc:237] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:40.669277 13800 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.670392 17242 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.670610 11511 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.671017 17242 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.672148 17242 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:40.673791 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6c1d3814e364e718449c3e4379b31ed with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.674206 11511 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:40.674185 13800 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:40.677079 11105 catalog_manager.cc:5484] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 5 to 6, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:40.679855 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c1d3814e364e718449c3e4379b31ed with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:40.680230 17513 tablet_replica.cc:330] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:40.680876 17513 raft_consensus.cc:2229] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.681099 17513 raft_consensus.cc:2258] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.682736 17513 ts_tablet_manager.cc:1875] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.685037 17513 ts_tablet_manager.cc:1888] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:40.685103 17513 log.cc:1196] T e6c1d3814e364e718449c3e4379b31ed P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/e6c1d3814e364e718449c3e4379b31ed
I20240501 19:56:40.685817 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet e6c1d3814e364e718449c3e4379b31ed (table rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]) successfully deleted
I20240501 19:56:40.689868 17524 ts_tablet_manager.cc:902] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:40.690778 17524 tablet_copy_client.cc:308] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:40.691267 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 00b7dd445e2846d5a0e1277e160d46bc from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:53821)
I20240501 19:56:40.691380 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 00b7dd445e2846d5a0e1277e160d46bc from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:53821: session id = 8125d8ad6d0d44caae9c5c90c90df96b-00b7dd445e2846d5a0e1277e160d46bc
I20240501 19:56:40.692327 11533 tablet_copy_source_session.cc:205] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:40.693535 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b7dd445e2846d5a0e1277e160d46bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.695286 17524 tablet_copy_client.cc:783] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:40.695390 17524 tablet_copy_client.cc:647] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:40.696525 17524 tablet_copy_client.cc:523] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:40.698366 17524 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:40.712523 11101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:40.717849 16657 tablet.cc:1722] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:40.720115 16333 tablet.cc:1722] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:40.721066 17428 tablet.cc:1722] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:40.726303 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=8d801b18020f4bc4a5c9deb4752973a5] alter complete (version 3)
I20240501 19:56:40.727963 17524 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:40.728246 17524 tablet_bootstrap.cc:492] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:40.728413 17524 ts_tablet_manager.cc:1367] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.003s
I20240501 19:56:40.728889 17524 raft_consensus.cc:348] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:40.729015 17524 raft_consensus.cc:729] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:40.729115 17524 consensus_queue.cc:260] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:40.729380 17524 ts_tablet_manager.cc:1398] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:40.729923 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-00b7dd445e2846d5a0e1277e160d46bc received from {username='slave'} at 127.5.40.5:53821
I20240501 19:56:40.730015 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-00b7dd445e2846d5a0e1277e160d46bc on tablet 00b7dd445e2846d5a0e1277e160d46bc with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:40.775660 11377 consensus_queue.cc:237] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:40.776989 13669 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:40.777051 11511 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:40.777391 16220 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:56:40.777602 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6326c660e6cb4eb289634a68fc50c4af. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:40.777612 17502 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:40.779482 17502 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:40.779771 11511 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:40.780383 13669 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:40.782802 11105 catalog_manager.cc:5484] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:40.790071 13669 consensus_queue.cc:237] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:40.791421 11377 raft_consensus.cc:1261] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:40.791568 13604 consensus_peers.cc:485] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): Couldn't send request to peer 8ad18cd54c6340fd92b874157875fa49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8b69cb1df2543f583eec2529a292897. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:40.791851 14062 raft_consensus.cc:1261] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.791853 17487 consensus_queue.cc:1035] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:40.792201 17487 consensus_queue.cc:1035] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:40.793888 17472 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:40.794442 14062 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:40.794898 11377 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:40.796909 11106 catalog_manager.cc:5484] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:40.826697 17611 ts_tablet_manager.cc:902] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:40.827195 17611 tablet_copy_client.cc:308] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:40.827680 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet c3b35dab7eec4b73b208f687dad46ba2 from peer 6921cf2e41f64d66842885512da8de6c ({username='slave'} at 127.5.40.7:41129)
I20240501 19:56:40.827791 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet c3b35dab7eec4b73b208f687dad46ba2 from peer 6921cf2e41f64d66842885512da8de6c at {username='slave'} at 127.5.40.7:41129: session id = 6921cf2e41f64d66842885512da8de6c-c3b35dab7eec4b73b208f687dad46ba2
I20240501 19:56:40.828505 11105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:40.828646 11105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:40.828676 11533 tablet_copy_source_session.cc:205] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:40.829702 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b35dab7eec4b73b208f687dad46ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.830896 13155 raft_consensus.cc:1050] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:40.831394 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 2dacd90dfac04a199914638f4a802259
I20240501 19:56:40.831367 13155 consensus_queue.cc:237] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:40.832242 14042 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dacd90dfac04a199914638f4a802259 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:40 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:40.832500 13910 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dacd90dfac04a199914638f4a802259 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:40 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:40.832854 17184 tablet_replica.cc:330] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:40.833019 17184 raft_consensus.cc:2229] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:40.833266 17184 raft_consensus.cc:2258] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.833273 17611 tablet_copy_client.cc:783] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:40.833417 17611 tablet_copy_client.cc:647] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:40.834317 11377 raft_consensus.cc:1261] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.834326 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dacd90dfac04a199914638f4a802259 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:40 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:40.834475 13669 raft_consensus.cc:1261] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.834740 17623 tablet_replica.cc:330] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:40.834934 17623 raft_consensus.cc:2229] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.835099 17623 raft_consensus.cc:2258] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.835316 13932 raft_consensus.cc:1203] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:40.835476 17611 tablet_copy_client.cc:523] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:40.836438 17623 ts_tablet_manager.cc:1875] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:40.834903 15765 consensus_queue.cc:1035] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.837016 17611 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:40.837796 17184 ts_tablet_manager.cc:1875] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:40.838362 15765 consensus_queue.cc:1035] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.838990 17623 ts_tablet_manager.cc:1888] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:40.839056 17623 log.cc:1196] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/2dacd90dfac04a199914638f4a802259
I20240501 19:56:40.839393 17623 ts_tablet_manager.cc:1909] T 2dacd90dfac04a199914638f4a802259 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:40.840258 17184 ts_tablet_manager.cc:1888] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:40.841724 17184 log.cc:1196] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/2dacd90dfac04a199914638f4a802259
I20240501 19:56:40.842351 11249 raft_consensus.cc:1261] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:40.842474 13437 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.842676 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 2dacd90dfac04a199914638f4a802259 (table rebalancer_dynamic_table_new_name [id=8d801b18020f4bc4a5c9deb4752973a5]) successfully deleted
I20240501 19:56:40.845793 16302 consensus_queue.cc:1035] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:40.845862 11380 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.846833 13669 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.847229 11249 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.848620 11105 catalog_manager.cc:5484] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:40.849180 17184 ts_tablet_manager.cc:1909] T 2dacd90dfac04a199914638f4a802259 P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:40.850796 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet 2dacd90dfac04a199914638f4a802259 (table rebalancer_dynamic_table_new_name [id=8d801b18020f4bc4a5c9deb4752973a5]) successfully deleted
I20240501 19:56:40.851367 17622 tablet_replica.cc:330] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:40.851619 17622 raft_consensus.cc:2229] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.851826 17622 raft_consensus.cc:2258] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.851980 17307 raft_consensus.cc:1050] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:40.852442 17307 consensus_queue.cc:237] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:40.852550 17622 ts_tablet_manager.cc:1875] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:40.854451 14062 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:40.854873 11644 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:40.855422 16220 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:40.856602 13930 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:40.856659 17622 ts_tablet_manager.cc:1888] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:40.856716 17622 log.cc:1196] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/2dacd90dfac04a199914638f4a802259
I20240501 19:56:40.857079 17622 ts_tablet_manager.cc:1909] T 2dacd90dfac04a199914638f4a802259 P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:40.857535 16220 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:40.859177 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet 2dacd90dfac04a199914638f4a802259 (table rebalancer_dynamic_table_new_name [id=8d801b18020f4bc4a5c9deb4752973a5]) successfully deleted
I20240501 19:56:40.859432 17502 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:40.866401 16220 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.868839 11644 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.869599 11101 catalog_manager.cc:5484] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:40.872984 13932 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.875239 17611 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:40.875600 17611 tablet_bootstrap.cc:492] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: Bootstrap complete.
I20240501 19:56:40.875797 17611 ts_tablet_manager.cc:1367] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.010s
I20240501 19:56:40.876255 17611 raft_consensus.cc:348] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:40.876367 17611 raft_consensus.cc:729] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: LEARNER
I20240501 19:56:40.876292 14062 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.876466 17611 consensus_queue.cc:260] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:40.876766 17611 ts_tablet_manager.cc:1398] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:40.878237 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session 6921cf2e41f64d66842885512da8de6c-c3b35dab7eec4b73b208f687dad46ba2 received from {username='slave'} at 127.5.40.7:41129
I20240501 19:56:40.878320 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session 6921cf2e41f64d66842885512da8de6c-c3b35dab7eec4b73b208f687dad46ba2 on tablet c3b35dab7eec4b73b208f687dad46ba2 with peer 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:40.879153 11377 consensus_queue.cc:237] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:40.880563 13932 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:40.880638 14062 raft_consensus.cc:1261] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:40.880918 17502 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:40.881389 16220 consensus_queue.cc:1035] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:40.882655 16220 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.883216 13932 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.886073 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01a10926bce14cf1bd12f21287e08c89 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:40.886312 11100 catalog_manager.cc:5484] T 01a10926bce14cf1bd12f21287e08c89 P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.886613 14062 raft_consensus.cc:2932] T 01a10926bce14cf1bd12f21287e08c89 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:40.889199 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 01a10926bce14cf1bd12f21287e08c89 with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:40.889456 17513 tablet_replica.cc:330] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:40.889612 17513 raft_consensus.cc:2229] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.889868 17513 raft_consensus.cc:2258] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.891405 17513 ts_tablet_manager.cc:1875] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:40.893985 17513 ts_tablet_manager.cc:1888] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:40.894049 17513 log.cc:1196] T 01a10926bce14cf1bd12f21287e08c89 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/01a10926bce14cf1bd12f21287e08c89
I20240501 19:56:40.894706 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 01a10926bce14cf1bd12f21287e08c89 (table rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]) successfully deleted
I20240501 19:56:40.933501 11100 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:40.939010 14042 tablet_service.cc:1467] Processing CreateTablet for tablet cda9ca49dc6d4e538e5e5e0a542412fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e75b354ce9d2422ba4adae3101fdcf33]), partition=
I20240501 19:56:40.939235 11359 tablet_service.cc:1467] Processing CreateTablet for tablet cda9ca49dc6d4e538e5e5e0a542412fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e75b354ce9d2422ba4adae3101fdcf33]), partition=
I20240501 19:56:40.939258 13910 tablet_service.cc:1467] Processing CreateTablet for tablet cda9ca49dc6d4e538e5e5e0a542412fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e75b354ce9d2422ba4adae3101fdcf33]), partition=
I20240501 19:56:40.939304 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda9ca49dc6d4e538e5e5e0a542412fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.939512 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda9ca49dc6d4e538e5e5e0a542412fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.939520 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda9ca49dc6d4e538e5e5e0a542412fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.941576 17253 tablet_bootstrap.cc:492] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:40.941644 17251 tablet_bootstrap.cc:492] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:40.942451 17253 tablet_bootstrap.cc:654] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.942466 17251 tablet_bootstrap.cc:654] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.943516 17253 tablet_bootstrap.cc:492] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:40.943603 17253 ts_tablet_manager.cc:1367] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:40.943686 17251 tablet_bootstrap.cc:492] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:40.943758 17251 ts_tablet_manager.cc:1367] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:40.944065 17253 raft_consensus.cc:348] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.944169 17253 raft_consensus.cc:374] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.944139 17251 raft_consensus.cc:348] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.944170 17687 tablet_bootstrap.cc:492] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:40.944198 17253 raft_consensus.cc:729] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.944221 17251 raft_consensus.cc:374] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.944249 17251 raft_consensus.cc:729] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.944330 17253 consensus_queue.cc:260] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.944353 17251 consensus_queue.cc:260] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.944592 17251 ts_tablet_manager.cc:1398] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:40.944599 17253 ts_tablet_manager.cc:1398] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.947499 17687 tablet_bootstrap.cc:654] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.949843 17687 tablet_bootstrap.cc:492] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:40.949951 17687 ts_tablet_manager.cc:1367] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:40.950398 17687 raft_consensus.cc:348] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.950552 17687 raft_consensus.cc:374] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.950608 17687 raft_consensus.cc:729] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.950771 17687 consensus_queue.cc:260] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:40.951035 17687 ts_tablet_manager.cc:1398] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.004492 17487 consensus_queue.cc:1035] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.008525 17472 consensus_queue.cc:1035] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.027274 17487 raft_consensus.cc:1050] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b: attempting to promote NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc to VOTER
I20240501 19:56:41.027771 17487 consensus_queue.cc:237] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } }
I20240501 19:56:41.029518 13932 raft_consensus.cc:1261] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:41.029745 11249 raft_consensus.cc:1261] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:41.029803 11511 raft_consensus.cc:1261] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:41.030146 15931 consensus_queue.cc:1035] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:41.030344 17472 consensus_queue.cc:1035] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:41.030642 15931 consensus_queue.cc:1035] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:41.032470 17487 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:41.033181 13932 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:41.034119 11249 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:41.034641 11511 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:41.037792 11101 catalog_manager.cc:5484] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:41.042430 13669 consensus_queue.cc:237] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } }
I20240501 19:56:41.044492 11249 raft_consensus.cc:1261] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:41.044996 15931 consensus_queue.cc:1035] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:41.045094 13932 raft_consensus.cc:1261] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:41.045898 15931 consensus_queue.cc:1035] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:41.047598 17472 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:41.049546 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 811d018fde314227a378f2d530e0a924 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:41.049988 11249 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:41.050034 11106 catalog_manager.cc:5484] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 3 to 4, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:41.052202 13932 raft_consensus.cc:2932] T 811d018fde314227a378f2d530e0a924 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:41.053784 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 811d018fde314227a378f2d530e0a924 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:41.054011 17623 tablet_replica.cc:330] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:41.054138 17623 raft_consensus.cc:2229] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.054270 17623 raft_consensus.cc:2258] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.055151 17623 ts_tablet_manager.cc:1875] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:41.057116 17623 ts_tablet_manager.cc:1888] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:41.057168 17623 log.cc:1196] T 811d018fde314227a378f2d530e0a924 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/811d018fde314227a378f2d530e0a924
I20240501 19:56:41.057652 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 811d018fde314227a378f2d530e0a924 (table rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]) successfully deleted
I20240501 19:56:41.124410 16341 consensus_queue.cc:1035] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.127575 16340 consensus_queue.cc:1035] T 313fb8dab2644b339cdd22647a8550fc P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.152616 11511 raft_consensus.cc:1203] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:41.171299 11106 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:41.178745 13780 tablet_service.cc:1467] Processing CreateTablet for tablet 512e3882ea4e4dbd97959c03a43de6d9 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1497c304da2b44bc874be05e8c72a7b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.179056 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512e3882ea4e4dbd97959c03a43de6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.179164 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 512e3882ea4e4dbd97959c03a43de6d9 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1497c304da2b44bc874be05e8c72a7b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.179491 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512e3882ea4e4dbd97959c03a43de6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.179730 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 512e3882ea4e4dbd97959c03a43de6d9 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1497c304da2b44bc874be05e8c72a7b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.179997 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512e3882ea4e4dbd97959c03a43de6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.181507 13781 tablet_service.cc:1467] Processing CreateTablet for tablet fc5eb3e030774199836cff2a657e8d56 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1497c304da2b44bc874be05e8c72a7b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.181744 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc5eb3e030774199836cff2a657e8d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.181738 17730 tablet_bootstrap.cc:492] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:41.182067 17732 tablet_bootstrap.cc:492] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:41.182798 13650 tablet_service.cc:1467] Processing CreateTablet for tablet fc5eb3e030774199836cff2a657e8d56 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1497c304da2b44bc874be05e8c72a7b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.182900 17730 tablet_bootstrap.cc:654] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.182924 17732 tablet_bootstrap.cc:654] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.183075 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc5eb3e030774199836cff2a657e8d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.183773 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 4acffebbd9284c6289361d23cea16b3e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1497c304da2b44bc874be05e8c72a7b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.184036 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acffebbd9284c6289361d23cea16b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.184098 14042 tablet_service.cc:1467] Processing CreateTablet for tablet 4acffebbd9284c6289361d23cea16b3e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1497c304da2b44bc874be05e8c72a7b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.184347 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acffebbd9284c6289361d23cea16b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.184378 13648 tablet_service.cc:1467] Processing CreateTablet for tablet 4acffebbd9284c6289361d23cea16b3e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1497c304da2b44bc874be05e8c72a7b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.184592 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acffebbd9284c6289361d23cea16b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.185401 14043 tablet_service.cc:1467] Processing CreateTablet for tablet fc5eb3e030774199836cff2a657e8d56 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=1497c304da2b44bc874be05e8c72a7b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.185599 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc5eb3e030774199836cff2a657e8d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.186599 17253 tablet_bootstrap.cc:492] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:41.187503 17253 tablet_bootstrap.cc:654] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.187737 17735 tablet_bootstrap.cc:492] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:41.187872 17731 tablet_bootstrap.cc:492] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:41.188107 17251 tablet_bootstrap.cc:492] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:41.188597 17735 tablet_bootstrap.cc:654] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.188935 17251 tablet_bootstrap.cc:654] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.189164 17731 tablet_bootstrap.cc:654] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.189889 17732 tablet_bootstrap.cc:492] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:41.189996 17732 ts_tablet_manager.cc:1367] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:41.190070 17253 tablet_bootstrap.cc:492] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:41.190158 17253 ts_tablet_manager.cc:1367] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:41.190505 17732 raft_consensus.cc:348] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:41.190577 17253 raft_consensus.cc:348] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.190636 17732 raft_consensus.cc:374] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.190667 17732 raft_consensus.cc:729] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.190668 17253 raft_consensus.cc:374] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.190706 17253 raft_consensus.cc:729] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.190809 17732 consensus_queue.cc:260] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:41.190824 17253 consensus_queue.cc:260] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.191087 17253 ts_tablet_manager.cc:1398] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:41.191087 17732 ts_tablet_manager.cc:1398] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:41.192893 17251 tablet_bootstrap.cc:492] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:41.192981 17251 ts_tablet_manager.cc:1367] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:41.193421 17735 tablet_bootstrap.cc:492] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:41.193437 17251 raft_consensus.cc:348] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.193504 17735 ts_tablet_manager.cc:1367] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:41.193532 17251 raft_consensus.cc:374] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.193560 17251 raft_consensus.cc:729] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.193662 17251 consensus_queue.cc:260] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.193917 17251 ts_tablet_manager.cc:1398] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.194064 17251 tablet_bootstrap.cc:492] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:41.194147 17735 raft_consensus.cc:348] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.194236 17735 raft_consensus.cc:374] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.194263 17735 raft_consensus.cc:729] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.194387 17735 consensus_queue.cc:260] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.195040 17251 tablet_bootstrap.cc:654] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.195637 17735 ts_tablet_manager.cc:1398] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:41.195793 17735 tablet_bootstrap.cc:492] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:41.197624 17735 tablet_bootstrap.cc:654] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.198896 17731 tablet_bootstrap.cc:492] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:41.199162 17731 ts_tablet_manager.cc:1367] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:41.199713 17251 tablet_bootstrap.cc:492] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:41.199792 17251 ts_tablet_manager.cc:1367] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:41.199846 17735 tablet_bootstrap.cc:492] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:41.199915 17735 ts_tablet_manager.cc:1367] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:41.200179 17251 raft_consensus.cc:348] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.200273 17251 raft_consensus.cc:374] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.200234 17735 raft_consensus.cc:348] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.200297 17251 raft_consensus.cc:729] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.200321 17735 raft_consensus.cc:374] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.200345 17735 raft_consensus.cc:729] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.200407 17251 consensus_queue.cc:260] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.200453 17735 consensus_queue.cc:260] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.200675 17251 ts_tablet_manager.cc:1398] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.200703 17735 ts_tablet_manager.cc:1398] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.201385 17731 raft_consensus.cc:348] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:41.201598 17731 raft_consensus.cc:374] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.201680 17731 raft_consensus.cc:729] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.201997 17731 consensus_queue.cc:260] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:41.203264 17731 ts_tablet_manager.cc:1398] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:41.203506 17731 tablet_bootstrap.cc:492] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:41.204479 17731 tablet_bootstrap.cc:654] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.206318 17747 ts_tablet_manager.cc:902] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: Initiating tablet copy from peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:41.206818 17747 tablet_copy_client.cc:308] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.5:42501
I20240501 19:56:41.207264 13690 tablet_copy_service.cc:140] P 8125d8ad6d0d44caae9c5c90c90df96b: Received BeginTabletCopySession request for tablet f8b69cb1df2543f583eec2529a292897 from peer 8ad18cd54c6340fd92b874157875fa49 ({username='slave'} at 127.5.40.4:35171)
I20240501 19:56:41.207355 13690 tablet_copy_service.cc:161] P 8125d8ad6d0d44caae9c5c90c90df96b: Beginning new tablet copy session on tablet f8b69cb1df2543f583eec2529a292897 from peer 8ad18cd54c6340fd92b874157875fa49 at {username='slave'} at 127.5.40.4:35171: session id = 8ad18cd54c6340fd92b874157875fa49-f8b69cb1df2543f583eec2529a292897
I20240501 19:56:41.208285 13690 tablet_copy_source_session.cc:205] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:41.208312 17730 tablet_bootstrap.cc:492] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:41.208402 17730 ts_tablet_manager.cc:1367] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240501 19:56:41.208911 17730 raft_consensus.cc:348] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:41.209022 17730 raft_consensus.cc:374] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.209026 17731 tablet_bootstrap.cc:492] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:41.209049 17730 raft_consensus.cc:729] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.209100 17731 ts_tablet_manager.cc:1367] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:41.209153 17730 consensus_queue.cc:260] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:41.209455 17730 ts_tablet_manager.cc:1398] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.209512 17731 raft_consensus.cc:348] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.209592 17731 raft_consensus.cc:374] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.209611 17731 raft_consensus.cc:729] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.209686 17731 consensus_queue.cc:260] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.209911 17731 ts_tablet_manager.cc:1398] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.210409 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b69cb1df2543f583eec2529a292897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.212430 17747 tablet_copy_client.cc:783] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:41.212626 17747 tablet_copy_client.cc:647] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:41.213640 17747 tablet_copy_client.cc:523] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:41.215014 17747 tablet_bootstrap.cc:492] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:41.215269 15931 raft_consensus.cc:482] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.215358 15931 raft_consensus.cc:504] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.215858 15931 leader_election.cc:288] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:41.216183 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc5eb3e030774199836cff2a657e8d56" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:41.216301 13799 raft_consensus.cc:2454] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:41.216276 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc5eb3e030774199836cff2a657e8d56" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:41.216403 14062 raft_consensus.cc:2454] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:41.216598 13606 leader_election.cc:302] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:41.216814 13155 consensus_queue.cc:1035] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.218173 15931 raft_consensus.cc:2781] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.218232 15931 raft_consensus.cc:482] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.218267 15931 raft_consensus.cc:3037] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.219035 15931 raft_consensus.cc:504] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.219378 15931 leader_election.cc:288] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:41.219838 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc5eb3e030774199836cff2a657e8d56" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:41.219955 13799 raft_consensus.cc:3037] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.220733 13799 raft_consensus.cc:2454] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:41.220960 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc5eb3e030774199836cff2a657e8d56" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:41.221076 14063 raft_consensus.cc:3037] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.221834 14063 raft_consensus.cc:2454] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:41.222100 13606 leader_election.cc:302] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:41.222575 15931 raft_consensus.cc:2781] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.222661 15931 raft_consensus.cc:686] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:41.222800 15931 consensus_queue.cc:237] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.224922 11100 catalog_manager.cc:5484] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:41.227423 15765 consensus_queue.cc:1035] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.229838 16220 raft_consensus.cc:482] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.229941 16220 raft_consensus.cc:504] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.230262 16220 leader_election.cc:288] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:41.231315 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda9ca49dc6d4e538e5e5e0a542412fe" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:41.231484 13932 raft_consensus.cc:2454] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:41.231804 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda9ca49dc6d4e538e5e5e0a542412fe" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:41.231935 14062 raft_consensus.cc:2454] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:41.232095 11313 leader_election.cc:302] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:41.232389 16220 raft_consensus.cc:2781] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.232446 16220 raft_consensus.cc:482] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.232481 16220 raft_consensus.cc:3037] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.233157 17747 tablet_bootstrap.cc:492] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:41.233350 17747 tablet_bootstrap.cc:492] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap complete.
I20240501 19:56:41.233474 17747 ts_tablet_manager.cc:1367] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240501 19:56:41.233498 16220 raft_consensus.cc:504] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.233819 16220 leader_election.cc:288] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:41.233971 17747 raft_consensus.cc:348] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:41.234082 17747 raft_consensus.cc:729] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: LEARNER
I20240501 19:56:41.234207 17747 consensus_queue.cc:260] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:41.234493 17747 ts_tablet_manager.cc:1398] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.234575 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda9ca49dc6d4e538e5e5e0a542412fe" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:41.234688 13932 raft_consensus.cc:3037] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.235548 13690 tablet_copy_service.cc:342] P 8125d8ad6d0d44caae9c5c90c90df96b: Request end of tablet copy session 8ad18cd54c6340fd92b874157875fa49-f8b69cb1df2543f583eec2529a292897 received from {username='slave'} at 127.5.40.4:35171
I20240501 19:56:41.235628 13690 tablet_copy_service.cc:433] P 8125d8ad6d0d44caae9c5c90c90df96b: ending tablet copy session 8ad18cd54c6340fd92b874157875fa49-f8b69cb1df2543f583eec2529a292897 on tablet f8b69cb1df2543f583eec2529a292897 with peer 8ad18cd54c6340fd92b874157875fa49
I20240501 19:56:41.235675 13932 raft_consensus.cc:2454] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:41.235884 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda9ca49dc6d4e538e5e5e0a542412fe" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:41.235944 11313 leader_election.cc:302] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:41.236001 14062 raft_consensus.cc:3037] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.236181 16220 raft_consensus.cc:2781] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.236274 16220 raft_consensus.cc:686] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:41.236395 16220 consensus_queue.cc:237] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.236974 14062 raft_consensus.cc:2454] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:41.238308 11105 catalog_manager.cc:5484] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:41.258163 13669 raft_consensus.cc:1203] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:41.262857 11644 consensus_queue.cc:237] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:41.264329 11508 raft_consensus.cc:1261] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:41.264711 11377 raft_consensus.cc:1261] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:41.264701 17777 consensus_queue.cc:1035] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:41.265071 17704 consensus_queue.cc:1035] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:41.265303 11577 consensus_peers.cc:485] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501): Couldn't send request to peer 8125d8ad6d0d44caae9c5c90c90df96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 858f60ea9d814c69ac626cd22c9ea994. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:41.266693 17777 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:41.267163 11508 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:41.269742 11106 catalog_manager.cc:5484] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:41.270817 11377 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } } }
I20240501 19:56:41.277561 16352 raft_consensus.cc:482] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.277683 16352 raft_consensus.cc:504] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:41.278155 16352 leader_election.cc:288] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:41.278600 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e3882ea4e4dbd97959c03a43de6d9" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:41.278726 11247 raft_consensus.cc:2454] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:41.278692 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e3882ea4e4dbd97959c03a43de6d9" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:41.278825 11508 raft_consensus.cc:2454] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:41.279043 13734 leader_election.cc:302] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:41.279317 16352 raft_consensus.cc:2781] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.279378 16352 raft_consensus.cc:482] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.279418 16352 raft_consensus.cc:3037] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.280480 16352 raft_consensus.cc:504] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:41.280750 11508 consensus_queue.cc:237] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:41.280915 16352 leader_election.cc:288] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:41.281311 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e3882ea4e4dbd97959c03a43de6d9" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:41.281420 11247 raft_consensus.cc:3037] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.282155 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e3882ea4e4dbd97959c03a43de6d9" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:41.282248 11508 raft_consensus.cc:3037] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.282439 11247 raft_consensus.cc:2454] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:41.282546 13669 raft_consensus.cc:1261] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:41.282846 13734 leader_election.cc:302] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:41.282997 16352 raft_consensus.cc:2781] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.283012 15765 consensus_queue.cc:1035] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:41.283078 16352 raft_consensus.cc:686] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:41.283203 11508 raft_consensus.cc:2454] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:41.283193 16352 consensus_queue.cc:237] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:41.283859 11247 raft_consensus.cc:1261] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:41.284754 15765 consensus_queue.cc:1035] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:41.285091 15765 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.285459 13669 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.285830 11106 catalog_manager.cc:5484] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:41.286440 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6f594c53151b4d00a1784b6366e2814a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:41.287037 17791 ts_tablet_manager.cc:902] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:41.288235 11105 catalog_manager.cc:5484] T 6f594c53151b4d00a1784b6366e2814a P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:41.288458 11247 raft_consensus.cc:2932] T 6f594c53151b4d00a1784b6366e2814a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.289765 17791 tablet_copy_client.cc:308] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.2:40185
I20240501 19:56:41.290797 11400 tablet_copy_service.cc:140] P ef60ab93b49c4346af4f62d3b98558b8: Received BeginTabletCopySession request for tablet 6326c660e6cb4eb289634a68fc50c4af from peer 96fc18dbb5fd41afb9bebf805e8d33a7 ({username='slave'} at 127.5.40.6:59475)
I20240501 19:56:41.290911 11400 tablet_copy_service.cc:161] P ef60ab93b49c4346af4f62d3b98558b8: Beginning new tablet copy session on tablet 6326c660e6cb4eb289634a68fc50c4af from peer 96fc18dbb5fd41afb9bebf805e8d33a7 at {username='slave'} at 127.5.40.6:59475: session id = 96fc18dbb5fd41afb9bebf805e8d33a7-6326c660e6cb4eb289634a68fc50c4af
I20240501 19:56:41.291383 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f594c53151b4d00a1784b6366e2814a with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:41.291942 11400 tablet_copy_source_session.cc:205] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:41.292523 17796 tablet_replica.cc:330] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:41.292667 17796 raft_consensus.cc:2229] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.292904 17796 raft_consensus.cc:2258] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.293141 17791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6326c660e6cb4eb289634a68fc50c4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.293538 17796 ts_tablet_manager.cc:1875] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:41.295571 17791 tablet_copy_client.cc:783] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:41.295596 17796 ts_tablet_manager.cc:1888] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:41.295745 17791 tablet_copy_client.cc:647] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:41.295764 17796 log.cc:1196] T 6f594c53151b4d00a1784b6366e2814a P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f594c53151b4d00a1784b6366e2814a
I20240501 19:56:41.296414 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 6f594c53151b4d00a1784b6366e2814a (table rebalance_test_table_15 [id=70b505231d114bdc86bbfec06b85baf4]) successfully deleted
I20240501 19:56:41.297901 17791 tablet_copy_client.cc:523] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:41.299329 17791 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:41.304210 17624 raft_consensus.cc:1050] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:41.304700 17624 consensus_queue.cc:237] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:41.306466 11508 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:41.306509 13932 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:41.306906 13669 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:41.306944 17805 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:41.307689 17033 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:41.307996 17805 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:41.310835 17033 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.311297 13669 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.314520 11508 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.314548 11105 catalog_manager.cc:5484] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.316303 13932 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.318045 11247 consensus_queue.cc:237] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:41.320379 13932 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:41.321086 13669 raft_consensus.cc:1261] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:41.321686 17804 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:41.322055 17804 consensus_queue.cc:1035] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:41.323825 17804 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.324002 13932 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.325531 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c07d9fc05254a6399afcc54b3efb8ac with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:41.326479 17791 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:41.326856 17791 tablet_bootstrap.cc:492] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap complete.
I20240501 19:56:41.326880 11106 catalog_manager.cc:5484] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:41.327231 17791 ts_tablet_manager.cc:1367] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20240501 19:56:41.327636 13669 raft_consensus.cc:2932] T 1c07d9fc05254a6399afcc54b3efb8ac P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.327780 17791 raft_consensus.cc:348] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:41.327883 17791 raft_consensus.cc:729] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: LEARNER
I20240501 19:56:41.328011 17791 consensus_queue.cc:260] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:41.328387 17791 ts_tablet_manager.cc:1398] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.328969 11400 tablet_copy_service.cc:342] P ef60ab93b49c4346af4f62d3b98558b8: Request end of tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-6326c660e6cb4eb289634a68fc50c4af received from {username='slave'} at 127.5.40.6:59475
I20240501 19:56:41.329056 11400 tablet_copy_service.cc:433] P ef60ab93b49c4346af4f62d3b98558b8: ending tablet copy session 96fc18dbb5fd41afb9bebf805e8d33a7-6326c660e6cb4eb289634a68fc50c4af on tablet 6326c660e6cb4eb289634a68fc50c4af with peer 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:41.330519 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c07d9fc05254a6399afcc54b3efb8ac with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:41.330734 17623 tablet_replica.cc:330] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:41.330875 17623 raft_consensus.cc:2229] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.331064 17623 raft_consensus.cc:2258] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.331817 17623 ts_tablet_manager.cc:1875] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:41.333931 17623 ts_tablet_manager.cc:1888] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:41.334007 17623 log.cc:1196] T 1c07d9fc05254a6399afcc54b3efb8ac P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c07d9fc05254a6399afcc54b3efb8ac
I20240501 19:56:41.334586 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 1c07d9fc05254a6399afcc54b3efb8ac (table rebalance_test_table_17 [id=a53582470d974a6197771e878781ae4f]) successfully deleted
I20240501 19:56:41.344578 11105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:41.348989 14062 raft_consensus.cc:1261] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.349010 13932 raft_consensus.cc:1261] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:41.349436 17502 consensus_queue.cc:1035] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.349586 16220 consensus_queue.cc:1035] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.353683 11100 catalog_manager.cc:5576] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789) does not have the latest schema for tablet cda9ca49dc6d4e538e5e5e0a542412fe (table rebalancer_dynamic_table [id=e75b354ce9d2422ba4adae3101fdcf33]). Expected version 1 got 0
I20240501 19:56:41.353672 17838 tablet.cc:1722] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:41.353987 17835 tablet.cc:1722] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:41.354219 11105 catalog_manager.cc:5576] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243) does not have the latest schema for tablet cda9ca49dc6d4e538e5e5e0a542412fe (table rebalancer_dynamic_table [id=e75b354ce9d2422ba4adae3101fdcf33]). Expected version 1 got 0
I20240501 19:56:41.354298 17835 tablet.cc:1620] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: MemRowSet was empty: no flush needed.
I20240501 19:56:41.354501 17838 tablet.cc:1620] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:41.354578 11106 catalog_manager.cc:5576] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185) does not have the latest schema for tablet cda9ca49dc6d4e538e5e5e0a542412fe (table rebalancer_dynamic_table [id=e75b354ce9d2422ba4adae3101fdcf33]). Expected version 1 got 0
I20240501 19:56:41.357699 11101 catalog_manager.cc:5576] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789) does not have the latest schema for tablet cda9ca49dc6d4e538e5e5e0a542412fe (table rebalancer_dynamic_table [id=e75b354ce9d2422ba4adae3101fdcf33]). Expected version 1 got 0
I20240501 19:56:41.358896 11105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=e75b354ce9d2422ba4adae3101fdcf33] alter complete (version 1)
I20240501 19:56:41.359922 17840 tablet.cc:1722] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:41.360381 17840 tablet.cc:1620] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:41.366696 13932 raft_consensus.cc:1203] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:41.430094 17448 raft_consensus.cc:482] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.430229 17448 raft_consensus.cc:504] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.430676 17448 leader_election.cc:288] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:41.431097 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acffebbd9284c6289361d23cea16b3e" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:41.431164 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acffebbd9284c6289361d23cea16b3e" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:41.431252 13667 raft_consensus.cc:2454] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:41.431279 13932 raft_consensus.cc:2454] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:41.431536 13996 leader_election.cc:302] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:41.431700 17448 raft_consensus.cc:2781] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.431758 17448 raft_consensus.cc:482] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.431803 17448 raft_consensus.cc:3037] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.432644 17448 raft_consensus.cc:504] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.432966 17448 leader_election.cc:288] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:41.433357 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acffebbd9284c6289361d23cea16b3e" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:41.433374 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acffebbd9284c6289361d23cea16b3e" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:41.433491 13932 raft_consensus.cc:3037] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.433501 13667 raft_consensus.cc:3037] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.434458 13667 raft_consensus.cc:2454] T 4acffebbd9284c6289361d23cea16b3e P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:41.434458 13932 raft_consensus.cc:2454] T 4acffebbd9284c6289361d23cea16b3e P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:41.434823 13996 leader_election.cc:302] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:41.435088 17448 raft_consensus.cc:2781] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.435178 17448 raft_consensus.cc:686] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:41.435317 17448 consensus_queue.cc:237] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.437480 11101 catalog_manager.cc:5484] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:41.461980 11101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:41.467417 17835 tablet.cc:1722] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:41.467789 17835 tablet.cc:1620] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: MemRowSet was empty: no flush needed.
I20240501 19:56:41.467944 17838 tablet.cc:1722] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:41.468212 17838 tablet.cc:1620] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:41.468487 17840 tablet.cc:1722] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:41.468760 17840 tablet.cc:1620] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:41.473309 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=e75b354ce9d2422ba4adae3101fdcf33] alter complete (version 2)
I20240501 19:56:41.546269 16341 raft_consensus.cc:1050] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c: attempting to promote NON_VOTER 08711534813d47ba85aaec0771ed5d0f to VOTER
I20240501 19:56:41.546751 16341 consensus_queue.cc:237] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:41.548372 14062 raft_consensus.cc:1261] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:41.548488 11508 raft_consensus.cc:1261] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:41.549105 17701 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:41.549278 13799 raft_consensus.cc:1261] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:41.549530 17701 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:41.549714 16340 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:41.551735 16340 raft_consensus.cc:2932] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:41.552170 11508 raft_consensus.cc:2932] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:41.553686 14062 raft_consensus.cc:2932] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:41.555066 11101 catalog_manager.cc:5484] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:41.559057 13932 consensus_queue.cc:237] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:41.560240 14062 raft_consensus.cc:1261] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:41.560433 11508 raft_consensus.cc:1261] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:41.560745 17873 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:41.561022 17873 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:41.562906 16340 raft_consensus.cc:2932] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:41.565057 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba2b4dda83bc408eaddec5ebc7b734d5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:41.565223 11508 raft_consensus.cc:2932] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:41.566159 11106 catalog_manager.cc:5484] T ba2b4dda83bc408eaddec5ebc7b734d5 P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:41.566666 14062 raft_consensus.cc:2932] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:41.569568 13781 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2b4dda83bc408eaddec5ebc7b734d5 with delete_type TABLET_DATA_TOMBSTONED (TS 96fc18dbb5fd41afb9bebf805e8d33a7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:41.570236 17877 tablet_replica.cc:330] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:41.570426 17877 raft_consensus.cc:2229] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.570621 17877 pending_rounds.cc:62] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Trying to abort 1 pending ops.
I20240501 19:56:41.570705 17877 pending_rounds.cc:68] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022974572734320640 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } } }
I20240501 19:56:41.570880 17877 raft_consensus.cc:2894] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 19:56:41.570953 17877 raft_consensus.cc:2866] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:41.571009 17877 raft_consensus.cc:2258] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.571611 17877 ts_tablet_manager.cc:1875] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:41.573396 17877 ts_tablet_manager.cc:1888] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:41.573460 17877 log.cc:1196] T ba2b4dda83bc408eaddec5ebc7b734d5 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/ba2b4dda83bc408eaddec5ebc7b734d5
I20240501 19:56:41.573983 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet ba2b4dda83bc408eaddec5ebc7b734d5 (table rebalancer_extra_table_0 [id=7bab06b5295f46fda40858755f9d58cc]) successfully deleted
I20240501 19:56:41.575562 11105 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:41.581020 17835 tablet.cc:1722] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:41.581923 17838 tablet.cc:1722] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:41.584226 17840 tablet.cc:1722] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:41.588243 11101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=e75b354ce9d2422ba4adae3101fdcf33] alter complete (version 3)
I20240501 19:56:41.690654 11100 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:41.690812 11100 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:41.692965 11100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet cda9ca49dc6d4e538e5e5e0a542412fe
I20240501 19:56:41.693560 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet cda9ca49dc6d4e538e5e5e0a542412fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:41 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:41.693604 13910 tablet_service.cc:1514] Processing DeleteTablet for tablet cda9ca49dc6d4e538e5e5e0a542412fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:41 UTC) from {username='slave'} at 127.0.0.1:42490
I20240501 19:56:41.693776 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet cda9ca49dc6d4e538e5e5e0a542412fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:41 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:41.693892 17934 tablet_replica.cc:330] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: stopping tablet replica
I20240501 19:56:41.693949 17796 tablet_replica.cc:330] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:41.693925 17933 tablet_replica.cc:330] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:41.694029 17934 raft_consensus.cc:2229] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.694074 17796 raft_consensus.cc:2229] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:41.694093 17933 raft_consensus.cc:2229] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.694147 17934 raft_consensus.cc:2258] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.694205 17933 raft_consensus.cc:2258] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.694273 17796 raft_consensus.cc:2258] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.694773 17934 ts_tablet_manager.cc:1875] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:41.694854 17796 ts_tablet_manager.cc:1875] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:41.694928 17933 ts_tablet_manager.cc:1875] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:41.696745 17796 ts_tablet_manager.cc:1888] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:41.696813 17796 log.cc:1196] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/cda9ca49dc6d4e538e5e5e0a542412fe
I20240501 19:56:41.697059 17796 ts_tablet_manager.cc:1909] T cda9ca49dc6d4e538e5e5e0a542412fe P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:41.697090 17934 ts_tablet_manager.cc:1888] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:41.697144 17933 ts_tablet_manager.cc:1888] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:41.697180 17934 log.cc:1196] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-6/wal/wals/cda9ca49dc6d4e538e5e5e0a542412fe
I20240501 19:56:41.697248 17933 log.cc:1196] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/cda9ca49dc6d4e538e5e5e0a542412fe
I20240501 19:56:41.697422 17934 ts_tablet_manager.cc:1909] T cda9ca49dc6d4e538e5e5e0a542412fe P 6921cf2e41f64d66842885512da8de6c: Deleting consensus metadata
I20240501 19:56:41.697495 17933 ts_tablet_manager.cc:1909] T cda9ca49dc6d4e538e5e5e0a542412fe P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:41.697728 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet cda9ca49dc6d4e538e5e5e0a542412fe (table rebalancer_dynamic_table_new_name [id=e75b354ce9d2422ba4adae3101fdcf33]) successfully deleted
I20240501 19:56:41.698104 11089 catalog_manager.cc:4830] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): tablet cda9ca49dc6d4e538e5e5e0a542412fe (table rebalancer_dynamic_table_new_name [id=e75b354ce9d2422ba4adae3101fdcf33]) successfully deleted
I20240501 19:56:41.698261 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet cda9ca49dc6d4e538e5e5e0a542412fe (table rebalancer_dynamic_table_new_name [id=e75b354ce9d2422ba4adae3101fdcf33]) successfully deleted
I20240501 19:56:41.709614 16352 consensus_queue.cc:1035] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.713462 14794 consensus_queue.cc:1035] T 512e3882ea4e4dbd97959c03a43de6d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.762652 13800 raft_consensus.cc:1203] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:41.765643 15931 consensus_queue.cc:1035] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.768707 15931 consensus_queue.cc:1035] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.783257 16302 raft_consensus.cc:1050] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:41.783680 16302 consensus_queue.cc:237] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:41.784974 13667 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:41.785087 13932 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:41.785185 11644 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:41.785405 13437 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:41.785580 13155 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:41.785755 16302 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:41.787595 16302 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.788002 11644 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.788187 13932 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.789626 13667 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.790958 11100 catalog_manager.cc:5484] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 7 to 8, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:41.794184 11508 consensus_queue.cc:237] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:41.794550 11101 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:41.795579 13932 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:41.796025 13667 raft_consensus.cc:1261] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:41.796020 16302 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:41.796504 16302 consensus_queue.cc:1035] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:41.798522 13155 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.799271 13932 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.799794 13667 raft_consensus.cc:2932] T 00b7dd445e2846d5a0e1277e160d46bc P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.800875 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00b7dd445e2846d5a0e1277e160d46bc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:41.802692 11101 catalog_manager.cc:5484] T 00b7dd445e2846d5a0e1277e160d46bc P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index 8 to 9, VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) evicted. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:41.803025 17947 ts_tablet_manager.cc:902] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:41.803543 17947 tablet_copy_client.cc:308] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:41.803568 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 92fbf530f7d6489c8def40821ffa7d83 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90ff28b408eb4e4e8b2ac577d1a8e685]), partition=
I20240501 19:56:41.803817 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fbf530f7d6489c8def40821ffa7d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.803854 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 92fbf530f7d6489c8def40821ffa7d83 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90ff28b408eb4e4e8b2ac577d1a8e685]), partition=
I20240501 19:56:41.804059 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fbf530f7d6489c8def40821ffa7d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.806111 17953 tablet_bootstrap.cc:492] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:41.806120 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet 858f60ea9d814c69ac626cd22c9ea994 from peer 8125d8ad6d0d44caae9c5c90c90df96b ({username='slave'} at 127.5.40.5:54335)
I20240501 19:56:41.806219 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet 858f60ea9d814c69ac626cd22c9ea994 from peer 8125d8ad6d0d44caae9c5c90c90df96b at {username='slave'} at 127.5.40.5:54335: session id = 8125d8ad6d0d44caae9c5c90c90df96b-858f60ea9d814c69ac626cd22c9ea994
I20240501 19:56:41.806277 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 92fbf530f7d6489c8def40821ffa7d83 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90ff28b408eb4e4e8b2ac577d1a8e685]), partition=
I20240501 19:56:41.806484 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fbf530f7d6489c8def40821ffa7d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.806906 17953 tablet_bootstrap.cc:654] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.807147 11664 tablet_copy_source_session.cc:205] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:41.807473 17952 tablet_bootstrap.cc:492] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:41.808091 17947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858f60ea9d814c69ac626cd22c9ea994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.808521 17952 tablet_bootstrap.cc:654] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.809078 17953 tablet_bootstrap.cc:492] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:41.809175 17953 ts_tablet_manager.cc:1367] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:41.809679 17953 raft_consensus.cc:348] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.810087 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b7dd445e2846d5a0e1277e160d46bc with delete_type TABLET_DATA_TOMBSTONED (TS 8ad18cd54c6340fd92b874157875fa49 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:41.810439 17956 tablet_replica.cc:330] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:41.809995 17953 raft_consensus.cc:374] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.810593 17953 raft_consensus.cc:729] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.810663 17956 raft_consensus.cc:2229] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.810806 17953 consensus_queue.cc:260] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.810868 17956 raft_consensus.cc:2258] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.811101 17953 ts_tablet_manager.cc:1398] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:41.812280 17956 ts_tablet_manager.cc:1875] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:41.812373 17952 tablet_bootstrap.cc:492] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:41.812470 17952 ts_tablet_manager.cc:1367] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:41.812929 17952 raft_consensus.cc:348] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.813047 17952 raft_consensus.cc:374] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.813087 17952 raft_consensus.cc:729] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.813225 17947 tablet_copy_client.cc:783] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:41.813215 17952 consensus_queue.cc:260] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.813373 17947 tablet_copy_client.cc:647] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:41.813489 17952 ts_tablet_manager.cc:1398] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.814450 17955 tablet_bootstrap.cc:492] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:41.815713 17955 tablet_bootstrap.cc:654] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.815804 17947 tablet_copy_client.cc:523] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:41.816874 17448 consensus_queue.cc:1035] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.817019 17947 tablet_bootstrap.cc:492] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:41.817818 17956 ts_tablet_manager.cc:1888] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:41.817893 17956 log.cc:1196] T 00b7dd445e2846d5a0e1277e160d46bc P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/00b7dd445e2846d5a0e1277e160d46bc
I20240501 19:56:41.818387 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 00b7dd445e2846d5a0e1277e160d46bc (table rebalance_test_table_3 [id=b21b5f3794704c0987b7b09cdfcfb7bd]) successfully deleted
I20240501 19:56:41.819922 11644 raft_consensus.cc:1203] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:41.822140 16195 consensus_queue.cc:1035] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:41.826259 17955 tablet_bootstrap.cc:492] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:41.826433 17955 ts_tablet_manager.cc:1367] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:56:41.827061 17955 raft_consensus.cc:348] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.827242 17955 raft_consensus.cc:374] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.827314 17955 raft_consensus.cc:729] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.827961 17955 consensus_queue.cc:260] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.829056 17955 ts_tablet_manager.cc:1398] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:41.837757 17947 tablet_bootstrap.cc:492] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:41.838024 17947 tablet_bootstrap.cc:492] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap complete.
I20240501 19:56:41.838178 17947 ts_tablet_manager.cc:1367] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240501 19:56:41.838536 17947 raft_consensus.cc:348] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:41.838639 17947 raft_consensus.cc:729] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: LEARNER
I20240501 19:56:41.838726 17947 consensus_queue.cc:260] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: NON_VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: true } }
I20240501 19:56:41.839002 17947 ts_tablet_manager.cc:1398] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:56:41.839495 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-858f60ea9d814c69ac626cd22c9ea994 received from {username='slave'} at 127.5.40.5:54335
I20240501 19:56:41.839576 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 8125d8ad6d0d44caae9c5c90c90df96b-858f60ea9d814c69ac626cd22c9ea994 on tablet 858f60ea9d814c69ac626cd22c9ea994 with peer 8125d8ad6d0d44caae9c5c90c90df96b
I20240501 19:56:41.891683 13437 raft_consensus.cc:1050] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER 6921cf2e41f64d66842885512da8de6c to VOTER
I20240501 19:56:41.892138 13437 consensus_queue.cc:237] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:41.893662 13800 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:41.893733 13932 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:41.893756 11247 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:41.894248 16302 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:41.894467 17978 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:41.894584 13155 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:41.896494 17978 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.897161 11247 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.898607 13932 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.899113 13800 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.900501 11105 catalog_manager.cc:5484] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 7 to 8, 6921cf2e41f64d66842885512da8de6c (127.5.40.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:41.928776 11508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2"
dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57188
I20240501 19:56:41.928921 11508 raft_consensus.cc:595] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:41.945991 11100 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:41.951843 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 72b93db52ae64bfbb274dbfc5d27eb86 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.952198 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b93db52ae64bfbb274dbfc5d27eb86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.952591 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 72b93db52ae64bfbb274dbfc5d27eb86 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.952971 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b93db52ae64bfbb274dbfc5d27eb86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.952672 13648 tablet_service.cc:1467] Processing CreateTablet for tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.953732 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.955221 17953 tablet_bootstrap.cc:492] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:41.956102 17953 tablet_bootstrap.cc:654] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.957010 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 72b93db52ae64bfbb274dbfc5d27eb86 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.957260 17953 tablet_bootstrap.cc:492] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:41.957288 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b93db52ae64bfbb274dbfc5d27eb86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.957341 17953 ts_tablet_manager.cc:1367] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:41.957432 16195 raft_consensus.cc:482] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:41.957513 16195 raft_consensus.cc:504] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.957769 17953 raft_consensus.cc:348] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.957854 17953 raft_consensus.cc:374] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.957880 16195 leader_election.cc:288] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:41.957898 17953 raft_consensus.cc:729] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.958009 17953 consensus_queue.cc:260] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.958267 17953 ts_tablet_manager.cc:1398] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.958462 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fbf530f7d6489c8def40821ffa7d83" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:41.958508 18018 tablet_bootstrap.cc:492] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:41.958580 11377 raft_consensus.cc:2454] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:41.958873 13998 leader_election.cc:302] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:41.959046 17448 raft_consensus.cc:2781] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:41.959085 17448 raft_consensus.cc:482] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:41.959116 17448 raft_consensus.cc:3037] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.959363 18018 tablet_bootstrap.cc:654] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.960072 18020 tablet_bootstrap.cc:492] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:41.960054 17448 raft_consensus.cc:504] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.960224 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.960377 17448 leader_election.cc:288] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:41.960451 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.960587 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fbf530f7d6489c8def40821ffa7d83" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:41.960719 11511 raft_consensus.cc:2454] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:41.961053 18020 tablet_bootstrap.cc:654] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.961444 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.961719 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.961797 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fbf530f7d6489c8def40821ffa7d83" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:41.961889 11511 raft_consensus.cc:3037] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.961921 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 3e16610fde0b47a199efc533627ae0ab (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.961894 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 3e16610fde0b47a199efc533627ae0ab (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.962124 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e16610fde0b47a199efc533627ae0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.962134 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e16610fde0b47a199efc533627ae0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.962800 11511 raft_consensus.cc:2454] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:41.963183 13996 leader_election.cc:302] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 533908e517b24f77b2af8b5863e4ae5a; no voters:
I20240501 19:56:41.963342 17448 raft_consensus.cc:2781] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:41.963415 17448 raft_consensus.cc:686] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:41.963557 17448 consensus_queue.cc:237] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:41.964159 17953 tablet_bootstrap.cc:492] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:41.964294 13648 tablet_service.cc:1467] Processing CreateTablet for tablet 3e16610fde0b47a199efc533627ae0ab (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:41.964560 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e16610fde0b47a199efc533627ae0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:41.964530 18022 tablet_bootstrap.cc:492] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:41.965054 17953 tablet_bootstrap.cc:654] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.965410 18022 tablet_bootstrap.cc:654] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.967806 18020 tablet_bootstrap.cc:492] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:41.967919 18020 ts_tablet_manager.cc:1367] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:41.968394 18020 raft_consensus.cc:348] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.968500 18020 raft_consensus.cc:374] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.968533 18020 raft_consensus.cc:729] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.968648 18020 consensus_queue.cc:260] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.968937 18020 ts_tablet_manager.cc:1398] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:41.969130 18020 tablet_bootstrap.cc:492] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:41.969173 18018 tablet_bootstrap.cc:492] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:41.969260 18018 ts_tablet_manager.cc:1367] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:41.969705 18018 raft_consensus.cc:348] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.969813 18018 raft_consensus.cc:374] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.969843 18018 raft_consensus.cc:729] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.969889 18022 tablet_bootstrap.cc:492] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:41.969980 18022 ts_tablet_manager.cc:1367] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:41.970012 18018 consensus_queue.cc:260] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.970099 18020 tablet_bootstrap.cc:654] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.970381 18018 ts_tablet_manager.cc:1398] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.970450 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fbf530f7d6489c8def40821ffa7d83" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:41.970535 18018 tablet_bootstrap.cc:492] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:41.970562 11377 raft_consensus.cc:3037] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:41.971424 18018 tablet_bootstrap.cc:654] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.971621 11377 raft_consensus.cc:2454] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:41.972116 17953 tablet_bootstrap.cc:492] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:41.972213 17953 ts_tablet_manager.cc:1367] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:41.972640 18020 tablet_bootstrap.cc:492] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:41.972715 18020 ts_tablet_manager.cc:1367] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:41.972692 17953 raft_consensus.cc:348] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:41.972792 17953 raft_consensus.cc:374] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.972820 17953 raft_consensus.cc:729] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.972929 17953 consensus_queue.cc:260] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:41.973181 17953 ts_tablet_manager.cc:1398] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.973135 18020 raft_consensus.cc:348] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.973227 18020 raft_consensus.cc:374] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.973259 18020 raft_consensus.cc:729] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.973433 18020 consensus_queue.cc:260] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.973753 18020 ts_tablet_manager.cc:1398] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:41.975286 11100 catalog_manager.cc:5484] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:41.976039 18022 raft_consensus.cc:348] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:41.976155 18022 raft_consensus.cc:374] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.976187 18022 raft_consensus.cc:729] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.976310 18022 consensus_queue.cc:260] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:41.976680 18019 tablet_bootstrap.cc:492] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:41.976702 18022 ts_tablet_manager.cc:1398] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:41.977789 18019 tablet_bootstrap.cc:654] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.979754 18018 tablet_bootstrap.cc:492] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:41.980015 18018 ts_tablet_manager.cc:1367] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240501 19:56:41.980509 18018 raft_consensus.cc:348] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.980600 18018 raft_consensus.cc:374] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.980635 18018 raft_consensus.cc:729] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.980775 18018 consensus_queue.cc:260] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.981024 18018 ts_tablet_manager.cc:1398] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.981673 18019 tablet_bootstrap.cc:492] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:41.981774 18019 ts_tablet_manager.cc:1367] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:41.982170 18019 raft_consensus.cc:348] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.982282 18019 raft_consensus.cc:374] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.982312 18019 raft_consensus.cc:729] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.982434 18019 consensus_queue.cc:260] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:41.982698 18019 ts_tablet_manager.cc:1398] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:41.982851 18019 tablet_bootstrap.cc:492] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:41.983704 18019 tablet_bootstrap.cc:654] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:41.986042 18019 tablet_bootstrap.cc:492] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:41.986130 18019 ts_tablet_manager.cc:1367] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:41.986508 18019 raft_consensus.cc:348] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:41.986595 18019 raft_consensus.cc:374] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:41.986620 18019 raft_consensus.cc:729] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:41.986722 18019 consensus_queue.cc:260] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:41.986969 18019 ts_tablet_manager.cc:1398] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:42.063382 17777 raft_consensus.cc:482] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.063576 17777 raft_consensus.cc:504] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:42.063972 17777 leader_election.cc:288] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:42.064266 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e16610fde0b47a199efc533627ae0ab" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:42.064386 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e16610fde0b47a199efc533627ae0ab" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:42.064427 14062 raft_consensus.cc:2454] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:42.064515 13669 raft_consensus.cc:2454] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:42.064788 11577 leader_election.cc:302] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:42.064962 17777 raft_consensus.cc:2781] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.065030 17777 raft_consensus.cc:482] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.065075 17777 raft_consensus.cc:3037] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.066206 17777 raft_consensus.cc:504] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:42.066565 17777 leader_election.cc:288] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:42.067059 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e16610fde0b47a199efc533627ae0ab" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:42.067191 13669 raft_consensus.cc:3037] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.067921 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e16610fde0b47a199efc533627ae0ab" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:42.068042 14062 raft_consensus.cc:3037] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.068151 13669 raft_consensus.cc:2454] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:42.068617 11577 leader_election.cc:302] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:42.068783 17946 raft_consensus.cc:2781] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.068885 17946 raft_consensus.cc:686] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:42.068958 14062 raft_consensus.cc:2454] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:42.069039 17946 consensus_queue.cc:237] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:42.070964 11101 catalog_manager.cc:5484] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:42.103416 11101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:42.107409 11511 raft_consensus.cc:1261] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.107535 11377 raft_consensus.cc:1261] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.107983 16195 consensus_queue.cc:1035] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.108143 17448 consensus_queue.cc:1035] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.111380 11101 catalog_manager.cc:5576] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965) does not have the latest schema for tablet 92fbf530f7d6489c8def40821ffa7d83 (table rebalancer_dynamic_table [id=90ff28b408eb4e4e8b2ac577d1a8e685]). Expected version 1 got 0
I20240501 19:56:42.111403 18035 tablet.cc:1722] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:42.111776 18035 tablet.cc:1620] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:42.111932 11105 catalog_manager.cc:5576] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185) does not have the latest schema for tablet 92fbf530f7d6489c8def40821ffa7d83 (table rebalancer_dynamic_table [id=90ff28b408eb4e4e8b2ac577d1a8e685]). Expected version 1 got 0
I20240501 19:56:42.114226 18036 tablet.cc:1722] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:42.116940 11101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=90ff28b408eb4e4e8b2ac577d1a8e685] alter complete (version 1)
I20240501 19:56:42.117462 11105 catalog_manager.cc:5576] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185) does not have the latest schema for tablet 92fbf530f7d6489c8def40821ffa7d83 (table rebalancer_dynamic_table [id=90ff28b408eb4e4e8b2ac577d1a8e685]). Expected version 1 got 0
I20240501 19:56:42.117660 18037 tablet.cc:1722] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:42.117806 18036 tablet.cc:1620] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: MemRowSet was empty: no flush needed.
I20240501 19:56:42.117964 18037 tablet.cc:1620] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:42.118074 11106 catalog_manager.cc:5576] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965) does not have the latest schema for tablet 92fbf530f7d6489c8def40821ffa7d83 (table rebalancer_dynamic_table [id=90ff28b408eb4e4e8b2ac577d1a8e685]). Expected version 1 got 0
I20240501 19:56:42.141053 15931 raft_consensus.cc:482] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.141182 15931 raft_consensus.cc:504] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:42.141624 15931 leader_election.cc:288] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:42.141928 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d0d6aaa47e4f7eaa80ec5e0207d51d" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:42.142021 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d0d6aaa47e4f7eaa80ec5e0207d51d" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:42.142094 13800 raft_consensus.cc:2454] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:42.142164 13932 raft_consensus.cc:2454] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:42.142479 13603 leader_election.cc:302] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:42.142693 15931 raft_consensus.cc:2781] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.142740 15931 raft_consensus.cc:482] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.142764 15931 raft_consensus.cc:3037] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.143711 15931 raft_consensus.cc:504] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:42.144045 15931 leader_election.cc:288] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:42.144428 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d0d6aaa47e4f7eaa80ec5e0207d51d" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:42.144479 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d0d6aaa47e4f7eaa80ec5e0207d51d" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:42.144536 13800 raft_consensus.cc:3037] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.144603 13932 raft_consensus.cc:3037] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.145466 13800 raft_consensus.cc:2454] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:42.145581 13932 raft_consensus.cc:2454] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:42.145777 13606 leader_election.cc:302] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:42.145949 15931 raft_consensus.cc:2781] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.146076 15931 raft_consensus.cc:686] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:42.146232 15931 consensus_queue.cc:237] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:42.148430 11106 catalog_manager.cc:5484] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:42.160399 11511 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2"
dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57210
I20240501 19:56:42.160532 11511 raft_consensus.cc:595] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:42.221748 11106 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:42.227339 18035 tablet.cc:1722] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:42.227699 18035 tablet.cc:1620] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:42.227738 18037 tablet.cc:1722] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:42.228006 18037 tablet.cc:1620] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:42.227901 18036 tablet.cc:1722] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:42.228268 18036 tablet.cc:1620] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: MemRowSet was empty: no flush needed.
I20240501 19:56:42.235073 11105 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=90ff28b408eb4e4e8b2ac577d1a8e685] alter complete (version 2)
I20240501 19:56:42.252192 16220 raft_consensus.cc:1050] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 to VOTER
I20240501 19:56:42.252600 16220 consensus_queue.cc:237] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:42.253794 13800 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:42.253893 11511 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:42.253911 13669 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:42.254192 18067 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:42.254359 16220 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:42.254540 17242 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:42.256553 16220 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:42.257011 13800 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:42.257961 13669 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:42.258190 11511 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:42.260396 11101 catalog_manager.cc:5484] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 7 to 8, 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:42.264398 11377 consensus_queue.cc:237] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } }
I20240501 19:56:42.265450 13669 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:42.265617 13800 raft_consensus.cc:1261] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:42.265808 18067 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:42.266168 17502 consensus_queue.cc:1035] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:42.267894 17502 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:42.268321 13800 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:42.268450 13667 raft_consensus.cc:2932] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:42.269348 11091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6326c660e6cb4eb289634a68fc50c4af with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:42.271118 11105 catalog_manager.cc:5484] T 6326c660e6cb4eb289634a68fc50c4af P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 8 to 9, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } }
I20240501 19:56:42.274231 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 6326c660e6cb4eb289634a68fc50c4af with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:42.274681 18071 tablet_replica.cc:330] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:42.274853 18071 raft_consensus.cc:2229] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:42.275100 18071 raft_consensus.cc:2258] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:42.275779 18071 ts_tablet_manager.cc:1875] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:42.277609 18071 ts_tablet_manager.cc:1888] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:42.277694 18071 log.cc:1196] T 6326c660e6cb4eb289634a68fc50c4af P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/6326c660e6cb4eb289634a68fc50c4af
I20240501 19:56:42.278297 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 6326c660e6cb4eb289634a68fc50c4af (table rebalance_test_table_11 [id=577228d32221400cbdb97b45554418ec]) successfully deleted
I20240501 19:56:42.293814 18064 raft_consensus.cc:482] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.293943 18064 raft_consensus.cc:504] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:42.294358 18064 leader_election.cc:288] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:42.294720 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b93db52ae64bfbb274dbfc5d27eb86" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:42.294746 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b93db52ae64bfbb274dbfc5d27eb86" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:42.294857 13800 raft_consensus.cc:2454] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:42.294898 13932 raft_consensus.cc:2454] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:42.295156 13999 leader_election.cc:302] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:42.295313 18064 raft_consensus.cc:2781] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.295377 18064 raft_consensus.cc:482] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.295413 18064 raft_consensus.cc:3037] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.296295 18064 raft_consensus.cc:504] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:42.296630 18064 leader_election.cc:288] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:42.296943 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b93db52ae64bfbb274dbfc5d27eb86" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:42.297057 13932 raft_consensus.cc:3037] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.297045 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b93db52ae64bfbb274dbfc5d27eb86" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:42.297156 13800 raft_consensus.cc:3037] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.298044 13932 raft_consensus.cc:2454] T 72b93db52ae64bfbb274dbfc5d27eb86 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:42.298192 13800 raft_consensus.cc:2454] T 72b93db52ae64bfbb274dbfc5d27eb86 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:42.298367 13996 leader_election.cc:302] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:42.298529 18064 raft_consensus.cc:2781] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.298625 18064 raft_consensus.cc:686] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:42.298759 18064 consensus_queue.cc:237] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:42.300635 11100 catalog_manager.cc:5484] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:42.337343 11100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:42.342681 18035 tablet.cc:1722] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:42.343219 18037 tablet.cc:1722] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:42.344274 18036 tablet.cc:1722] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:42.349223 11100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=90ff28b408eb4e4e8b2ac577d1a8e685] alter complete (version 3)
I20240501 19:56:42.351049 13437 raft_consensus.cc:979] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: : Instructing follower 10ee60ead7b94c93ab749ec16a04b0dc to start an election
I20240501 19:56:42.351181 16302 raft_consensus.cc:1067] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Signalling peer 10ee60ead7b94c93ab749ec16a04b0dc to start an election
I20240501 19:56:42.352438 11247 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2"
dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
from {username='slave'} at 127.5.40.3:56253
I20240501 19:56:42.352613 11247 raft_consensus.cc:482] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:42.352689 11247 raft_consensus.cc:3037] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:42.353649 11247 raft_consensus.cc:504] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:42.354068 11247 leader_election.cc:288] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:42.354327 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:42.354458 11511 raft_consensus.cc:3032] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:42.354517 11511 raft_consensus.cc:729] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:42.354526 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:42.354589 13932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b35dab7eec4b73b208f687dad46ba2" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:42.354630 13800 raft_consensus.cc:3037] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:42.354687 13932 raft_consensus.cc:3037] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:42.354657 11511 consensus_queue.cc:260] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:42.354841 11511 raft_consensus.cc:3037] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:42.355388 13932 raft_consensus.cc:2454] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 2.
I20240501 19:56:42.355777 13800 raft_consensus.cc:2454] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 2.
I20240501 19:56:42.355788 11511 raft_consensus.cc:2454] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 2.
I20240501 19:56:42.356073 11184 leader_election.cc:302] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:42.356305 17033 raft_consensus.cc:2781] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:42.356448 17033 raft_consensus.cc:686] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:42.356597 17033 consensus_queue.cc:237] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:42.357678 13667 raft_consensus.cc:1203] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:42.359609 11100 catalog_manager.cc:5484] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 1 to 2, leader changed from 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 2 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:42.413726 15931 raft_consensus.cc:1050] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b: attempting to promote NON_VOTER 8ad18cd54c6340fd92b874157875fa49 to VOTER
I20240501 19:56:42.414106 15931 consensus_queue.cc:237] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:42.415366 11644 raft_consensus.cc:1261] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:42.415382 14062 raft_consensus.cc:1261] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:42.415568 11377 raft_consensus.cc:1261] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:42.415808 17696 consensus_queue.cc:1035] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:42.416021 18112 consensus_queue.cc:1035] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:42.416190 15931 consensus_queue.cc:1035] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:42.418080 15931 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:42.418608 11644 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:42.420096 14062 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:42.420194 11377 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:42.422468 11100 catalog_manager.cc:5484] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:42.425845 13667 consensus_queue.cc:237] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:42.426919 11644 raft_consensus.cc:1261] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:42.427139 14062 raft_consensus.cc:1261] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:42.427306 18112 consensus_queue.cc:1035] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:42.427562 17696 consensus_queue.cc:1035] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:42.429257 18112 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:42.429888 11644 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:42.431165 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8b69cb1df2543f583eec2529a292897 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:42.431214 14062 raft_consensus.cc:2932] T f8b69cb1df2543f583eec2529a292897 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:42.432408 11101 catalog_manager.cc:5484] T f8b69cb1df2543f583eec2529a292897 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 3 to 4, VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) evicted. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:42.435667 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet f8b69cb1df2543f583eec2529a292897 with delete_type TABLET_DATA_TOMBSTONED (TS ef60ab93b49c4346af4f62d3b98558b8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:42.436053 18116 tablet_replica.cc:330] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:42.436195 18116 raft_consensus.cc:2229] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:42.436350 18116 raft_consensus.cc:2258] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:42.437000 18116 ts_tablet_manager.cc:1875] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:42.438941 18116 ts_tablet_manager.cc:1888] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:42.439011 18116 log.cc:1196] T f8b69cb1df2543f583eec2529a292897 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/f8b69cb1df2543f583eec2529a292897
I20240501 19:56:42.439569 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet f8b69cb1df2543f583eec2529a292897 (table rebalancer_extra_table_10 [id=0a06c3c05b48472b9fa8346b606e7e19]) successfully deleted
I20240501 19:56:42.452363 11105 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:42.452533 11105 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:42.454631 11105 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 92fbf530f7d6489c8def40821ffa7d83
I20240501 19:56:42.455293 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 92fbf530f7d6489c8def40821ffa7d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:42 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:42.455293 11359 tablet_service.cc:1514] Processing DeleteTablet for tablet 92fbf530f7d6489c8def40821ffa7d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:42 UTC) from {username='slave'} at 127.0.0.1:35738
I20240501 19:56:42.455494 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet 92fbf530f7d6489c8def40821ffa7d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:42 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:42.455535 18116 tablet_replica.cc:330] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: stopping tablet replica
I20240501 19:56:42.455540 18071 tablet_replica.cc:330] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:42.455664 18071 raft_consensus.cc:2229] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:42.455664 18116 raft_consensus.cc:2229] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:42.455796 18116 raft_consensus.cc:2258] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:42.455799 18071 raft_consensus.cc:2258] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:42.455914 18117 tablet_replica.cc:330] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:42.456058 18117 raft_consensus.cc:2229] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:42.456323 18117 raft_consensus.cc:2258] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:42.456465 18116 ts_tablet_manager.cc:1875] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:42.456460 18071 ts_tablet_manager.cc:1875] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:42.457029 18117 ts_tablet_manager.cc:1875] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:42.458685 18116 ts_tablet_manager.cc:1888] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:42.458686 18071 ts_tablet_manager.cc:1888] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:42.458770 18116 log.cc:1196] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-1/wal/wals/92fbf530f7d6489c8def40821ffa7d83
I20240501 19:56:42.458770 18071 log.cc:1196] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/92fbf530f7d6489c8def40821ffa7d83
I20240501 19:56:42.459029 18071 ts_tablet_manager.cc:1909] T 92fbf530f7d6489c8def40821ffa7d83 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:42.459043 18116 ts_tablet_manager.cc:1909] T 92fbf530f7d6489c8def40821ffa7d83 P ef60ab93b49c4346af4f62d3b98558b8: Deleting consensus metadata
I20240501 19:56:42.459199 18117 ts_tablet_manager.cc:1888] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:42.459300 18117 log.cc:1196] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/92fbf530f7d6489c8def40821ffa7d83
I20240501 19:56:42.459585 18117 ts_tablet_manager.cc:1909] T 92fbf530f7d6489c8def40821ffa7d83 P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:42.459977 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 92fbf530f7d6489c8def40821ffa7d83 (table rebalancer_dynamic_table_new_name [id=90ff28b408eb4e4e8b2ac577d1a8e685]) successfully deleted
I20240501 19:56:42.459957 11091 catalog_manager.cc:4830] TS ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): tablet 92fbf530f7d6489c8def40821ffa7d83 (table rebalancer_dynamic_table_new_name [id=90ff28b408eb4e4e8b2ac577d1a8e685]) successfully deleted
I20240501 19:56:42.460184 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet 92fbf530f7d6489c8def40821ffa7d83 (table rebalancer_dynamic_table_new_name [id=90ff28b408eb4e4e8b2ac577d1a8e685]) successfully deleted
I20240501 19:56:42.556365 11105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:42.561507 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 875f9bdd03e74522b95d9585d325bf23 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=333de63ff3cc4460ab6911a2153bac92]), partition=
I20240501 19:56:42.561664 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 875f9bdd03e74522b95d9585d325bf23 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=333de63ff3cc4460ab6911a2153bac92]), partition=
I20240501 19:56:42.561821 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875f9bdd03e74522b95d9585d325bf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.561964 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875f9bdd03e74522b95d9585d325bf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.564172 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 875f9bdd03e74522b95d9585d325bf23 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=333de63ff3cc4460ab6911a2153bac92]), partition=
I20240501 19:56:42.564292 18118 tablet_bootstrap.cc:492] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:42.564432 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875f9bdd03e74522b95d9585d325bf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.565166 18118 tablet_bootstrap.cc:654] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.566206 18118 tablet_bootstrap.cc:492] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:42.566294 18118 ts_tablet_manager.cc:1367] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:42.566596 18121 tablet_bootstrap.cc:492] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:42.566768 18119 tablet_bootstrap.cc:492] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:42.566808 18118 raft_consensus.cc:348] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.566955 18118 raft_consensus.cc:374] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.567008 18118 raft_consensus.cc:729] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.567164 18118 consensus_queue.cc:260] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.567520 18121 tablet_bootstrap.cc:654] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.567600 18118 ts_tablet_manager.cc:1398] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.567922 18119 tablet_bootstrap.cc:654] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.568743 18121 tablet_bootstrap.cc:492] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:42.568840 18121 ts_tablet_manager.cc:1367] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:42.569124 18119 tablet_bootstrap.cc:492] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:42.569262 18119 ts_tablet_manager.cc:1367] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:42.569324 18121 raft_consensus.cc:348] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.569445 18121 raft_consensus.cc:374] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.569475 18121 raft_consensus.cc:729] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.569612 18121 consensus_queue.cc:260] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.569783 18119 raft_consensus.cc:348] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.569970 18119 raft_consensus.cc:374] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.570089 18119 raft_consensus.cc:729] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.570299 18119 consensus_queue.cc:260] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.570499 18121 ts_tablet_manager.cc:1398] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:42.570734 18119 ts_tablet_manager.cc:1398] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.577410 17777 consensus_queue.cc:1035] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.580221 18126 consensus_queue.cc:1035] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.669243 18073 raft_consensus.cc:482] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.669381 18073 raft_consensus.cc:504] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.669795 18073 leader_election.cc:288] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:42.670315 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f9bdd03e74522b95d9585d325bf23" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:42.670315 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f9bdd03e74522b95d9585d325bf23" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:42.670462 11511 raft_consensus.cc:2454] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:42.670462 13800 raft_consensus.cc:2454] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:42.670735 13996 leader_election.cc:302] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 533908e517b24f77b2af8b5863e4ae5a; no voters:
I20240501 19:56:42.670895 18073 raft_consensus.cc:2781] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.670956 18073 raft_consensus.cc:482] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.671010 18073 raft_consensus.cc:3037] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.672144 18073 raft_consensus.cc:504] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.672442 18073 leader_election.cc:288] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:42.672957 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f9bdd03e74522b95d9585d325bf23" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:42.672963 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f9bdd03e74522b95d9585d325bf23" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:42.673069 11511 raft_consensus.cc:3037] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.673069 13800 raft_consensus.cc:3037] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.674010 13800 raft_consensus.cc:2454] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:42.674010 11511 raft_consensus.cc:2454] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:42.674293 13996 leader_election.cc:302] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 533908e517b24f77b2af8b5863e4ae5a; no voters:
I20240501 19:56:42.674475 18073 raft_consensus.cc:2781] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.674572 18073 raft_consensus.cc:686] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:42.674736 18073 consensus_queue.cc:237] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.675087 15931 consensus_queue.cc:1035] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.677498 11101 catalog_manager.cc:5484] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:42.679580 15931 consensus_queue.cc:1035] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.783427 17946 raft_consensus.cc:1050] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 8125d8ad6d0d44caae9c5c90c90df96b to VOTER
I20240501 19:56:42.783823 17946 consensus_queue.cc:237] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:42.785074 11377 raft_consensus.cc:1261] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:42.785152 11511 raft_consensus.cc:1261] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:42.785265 13669 raft_consensus.cc:1261] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:42.785558 18160 consensus_queue.cc:1035] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:42.785720 18126 consensus_queue.cc:1035] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:42.785918 17946 consensus_queue.cc:1035] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:42.787695 17946 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:42.788148 11511 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:42.788441 13669 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:42.789851 11377 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:42.790534 11105 catalog_manager.cc:5484] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 4 to 5, 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:42.792940 11106 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:42.794526 11644 consensus_queue.cc:237] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:42.796658 11377 raft_consensus.cc:1261] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:42.797722 18160 consensus_queue.cc:1035] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:42.798518 13669 raft_consensus.cc:1261] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:42.798758 13800 raft_consensus.cc:1261] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.799125 18073 consensus_queue.cc:1035] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.799463 11511 raft_consensus.cc:1261] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.799870 18073 consensus_queue.cc:1035] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.799553 18160 consensus_queue.cc:1035] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:42.800839 17777 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:42.803231 11100 catalog_manager.cc:5576] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331) does not have the latest schema for tablet 875f9bdd03e74522b95d9585d325bf23 (table rebalancer_dynamic_table [id=333de63ff3cc4460ab6911a2153bac92]). Expected version 1 got 0
I20240501 19:56:42.803133 11377 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:42.803987 11101 catalog_manager.cc:5576] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965) does not have the latest schema for tablet 875f9bdd03e74522b95d9585d325bf23 (table rebalancer_dynamic_table [id=333de63ff3cc4460ab6911a2153bac92]). Expected version 1 got 0
I20240501 19:56:42.804021 18035 tablet.cc:1722] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:42.804343 18035 tablet.cc:1620] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:42.805030 11106 catalog_manager.cc:5484] T 858f60ea9d814c69ac626cd22c9ea994 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:42.806319 18037 tablet.cc:1722] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:42.806589 18037 tablet.cc:1620] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:42.808017 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 858f60ea9d814c69ac626cd22c9ea994 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:42.808383 11105 catalog_manager.cc:5576] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331) does not have the latest schema for tablet 875f9bdd03e74522b95d9585d325bf23 (table rebalancer_dynamic_table [id=333de63ff3cc4460ab6911a2153bac92]). Expected version 1 got 0
I20240501 19:56:42.807672 18166 tablet.cc:1722] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:42.808907 18166 tablet.cc:1620] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:42.810282 13930 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:42.811343 17033 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:42.811730 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 858f60ea9d814c69ac626cd22c9ea994 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:42.811924 18071 tablet_replica.cc:330] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:42.812067 18071 raft_consensus.cc:2229] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:42.811865 11101 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=333de63ff3cc4460ab6911a2153bac92] alter complete (version 1)
I20240501 19:56:42.812243 18071 raft_consensus.cc:2258] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:42.812795 13669 raft_consensus.cc:2932] T 858f60ea9d814c69ac626cd22c9ea994 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } } }
I20240501 19:56:42.815012 18071 ts_tablet_manager.cc:1875] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:42.817173 18071 ts_tablet_manager.cc:1888] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:42.817252 18071 log.cc:1196] T 858f60ea9d814c69ac626cd22c9ea994 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/858f60ea9d814c69ac626cd22c9ea994
I20240501 19:56:42.817665 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 858f60ea9d814c69ac626cd22c9ea994 (table rebalance_test_table_5 [id=c74ce2ff84304168968af98e2d8fb508]) successfully deleted
I20240501 19:56:42.833823 11105 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:42.840692 13910 tablet_service.cc:1467] Processing CreateTablet for tablet b85f5acd65914c2d8f1cc48d668c40eb (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=a9d99d04ee18480a95c8eac9ee31056d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.841015 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85f5acd65914c2d8f1cc48d668c40eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.841344 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 85787888b78240b9b064a8e230c07681 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=a9d99d04ee18480a95c8eac9ee31056d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.841606 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85787888b78240b9b064a8e230c07681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.841594 11227 tablet_service.cc:1467] Processing CreateTablet for tablet 200886108633404e917f52c7db2a18d9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=a9d99d04ee18480a95c8eac9ee31056d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.841885 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200886108633404e917f52c7db2a18d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.843693 11229 tablet_service.cc:1467] Processing CreateTablet for tablet 85787888b78240b9b064a8e230c07681 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=a9d99d04ee18480a95c8eac9ee31056d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.843818 18167 tablet_bootstrap.cc:492] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:42.843926 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85787888b78240b9b064a8e230c07681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.844115 13648 tablet_service.cc:1467] Processing CreateTablet for tablet b85f5acd65914c2d8f1cc48d668c40eb (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=a9d99d04ee18480a95c8eac9ee31056d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.844358 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85f5acd65914c2d8f1cc48d668c40eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.844728 18167 tablet_bootstrap.cc:654] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.844825 18168 tablet_bootstrap.cc:492] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:42.845835 18168 tablet_bootstrap.cc:654] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.846462 11359 tablet_service.cc:1467] Processing CreateTablet for tablet b85f5acd65914c2d8f1cc48d668c40eb (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=a9d99d04ee18480a95c8eac9ee31056d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.846745 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85f5acd65914c2d8f1cc48d668c40eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.846827 18167 tablet_bootstrap.cc:492] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:42.846829 11358 tablet_service.cc:1467] Processing CreateTablet for tablet 85787888b78240b9b064a8e230c07681 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=a9d99d04ee18480a95c8eac9ee31056d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.846874 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 200886108633404e917f52c7db2a18d9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=a9d99d04ee18480a95c8eac9ee31056d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.846961 18167 ts_tablet_manager.cc:1367] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:42.847087 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85787888b78240b9b064a8e230c07681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.847126 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200886108633404e917f52c7db2a18d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.847404 18167 raft_consensus.cc:348] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:42.847523 18167 raft_consensus.cc:374] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.847550 18167 raft_consensus.cc:729] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.847662 18167 consensus_queue.cc:260] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:42.848001 18167 ts_tablet_manager.cc:1398] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.848979 18171 tablet_bootstrap.cc:492] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:42.849210 18118 tablet_bootstrap.cc:492] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:42.849893 18171 tablet_bootstrap.cc:654] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.850068 18118 tablet_bootstrap.cc:654] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.850781 18168 tablet_bootstrap.cc:492] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:42.850868 18168 ts_tablet_manager.cc:1367] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:42.850879 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 200886108633404e917f52c7db2a18d9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=a9d99d04ee18480a95c8eac9ee31056d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.851150 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200886108633404e917f52c7db2a18d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.851294 18168 raft_consensus.cc:348] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.851388 18168 raft_consensus.cc:374] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.852278 18118 tablet_bootstrap.cc:492] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:42.852378 18118 ts_tablet_manager.cc:1367] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:42.851418 18168 raft_consensus.cc:729] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.852838 18171 tablet_bootstrap.cc:492] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:42.852789 18118 raft_consensus.cc:348] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.852898 18118 raft_consensus.cc:374] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.852849 18168 consensus_queue.cc:260] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.852918 18171 ts_tablet_manager.cc:1367] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:42.852931 18118 raft_consensus.cc:729] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.853039 18118 consensus_queue.cc:260] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.853149 18168 ts_tablet_manager.cc:1398] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.853299 18118 ts_tablet_manager.cc:1398] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.853317 18168 tablet_bootstrap.cc:492] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:42.853610 18171 raft_consensus.cc:348] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:42.853708 18171 raft_consensus.cc:374] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.853713 18121 tablet_bootstrap.cc:492] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:42.853736 18171 raft_consensus.cc:729] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.853842 18171 consensus_queue.cc:260] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:42.854427 18168 tablet_bootstrap.cc:654] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.854494 18121 tablet_bootstrap.cc:654] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.856848 18064 consensus_queue.cc:1035] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.857029 18171 ts_tablet_manager.cc:1398] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:42.859390 18171 tablet_bootstrap.cc:492] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:42.860642 18171 tablet_bootstrap.cc:654] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.861717 18171 tablet_bootstrap.cc:492] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:42.861827 18171 ts_tablet_manager.cc:1367] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:42.862246 18171 raft_consensus.cc:348] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:42.862442 18171 raft_consensus.cc:374] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.862519 18171 raft_consensus.cc:729] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.862735 18171 consensus_queue.cc:260] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:42.863032 18171 ts_tablet_manager.cc:1398] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:42.863126 17448 consensus_queue.cc:1035] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.867166 18121 tablet_bootstrap.cc:492] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:42.867265 18121 ts_tablet_manager.cc:1367] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240501 19:56:42.867720 18121 raft_consensus.cc:348] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.867830 18121 raft_consensus.cc:374] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.867863 18121 raft_consensus.cc:729] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.867986 18121 consensus_queue.cc:260] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:42.868247 18121 ts_tablet_manager.cc:1398] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.868875 18170 tablet_bootstrap.cc:492] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:42.869063 18168 tablet_bootstrap.cc:492] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc: No bootstrap required, opened a new log
I20240501 19:56:42.869153 18168 ts_tablet_manager.cc:1367] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:56:42.869616 18168 raft_consensus.cc:348] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:42.869714 18168 raft_consensus.cc:374] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.869743 18168 raft_consensus.cc:729] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.869836 18170 tablet_bootstrap.cc:654] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.869886 18168 consensus_queue.cc:260] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:42.870157 18168 ts_tablet_manager.cc:1398] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:42.872253 11511 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:42.872613 17033 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:42.875629 13800 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 2 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:42.876624 17033 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:42.881311 18170 tablet_bootstrap.cc:492] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:42.881500 18170 ts_tablet_manager.cc:1367] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 19:56:42.883531 18170 raft_consensus.cc:348] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:42.883652 18170 raft_consensus.cc:374] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.883718 18170 raft_consensus.cc:729] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.883873 18170 consensus_queue.cc:260] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:42.884179 18170 ts_tablet_manager.cc:1398] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:42.885780 18170 tablet_bootstrap.cc:492] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:42.886745 18170 tablet_bootstrap.cc:654] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.887775 18170 tablet_bootstrap.cc:492] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:42.887854 18170 ts_tablet_manager.cc:1367] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:42.888401 18170 raft_consensus.cc:348] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:42.888574 18170 raft_consensus.cc:374] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.888653 18170 raft_consensus.cc:729] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.888813 18170 consensus_queue.cc:260] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:42.889156 18170 ts_tablet_manager.cc:1398] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.917944 11100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:42.922945 18035 tablet.cc:1722] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:42.923305 18035 tablet.cc:1620] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:42.923499 18037 tablet.cc:1722] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:42.923590 18166 tablet.cc:1722] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:42.923796 18037 tablet.cc:1620] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:42.923879 18166 tablet.cc:1620] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:42.928743 11100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=333de63ff3cc4460ab6911a2153bac92] alter complete (version 2)
I20240501 19:56:43.030815 11100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:43.036191 18035 tablet.cc:1722] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:43.036878 18166 tablet.cc:1722] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:43.037842 18037 tablet.cc:1722] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:43.043519 11100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=333de63ff3cc4460ab6911a2153bac92] alter complete (version 3)
I20240501 19:56:43.121028 18185 raft_consensus.cc:482] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.121168 18185 raft_consensus.cc:504] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:43.121631 18185 leader_election.cc:288] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:43.122047 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85f5acd65914c2d8f1cc48d668c40eb" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:43.122035 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85f5acd65914c2d8f1cc48d668c40eb" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:43.122192 11377 raft_consensus.cc:2454] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:43.122198 13669 raft_consensus.cc:2454] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:43.122521 13867 leader_election.cc:302] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:43.122696 18185 raft_consensus.cc:2781] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.122766 18185 raft_consensus.cc:482] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.122805 18185 raft_consensus.cc:3037] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.123872 18185 raft_consensus.cc:504] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:43.124187 18185 leader_election.cc:288] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:43.124495 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85f5acd65914c2d8f1cc48d668c40eb" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:43.124562 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85f5acd65914c2d8f1cc48d668c40eb" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:43.124615 11377 raft_consensus.cc:3037] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.124691 13669 raft_consensus.cc:3037] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.125608 11377 raft_consensus.cc:2454] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:43.125707 13669 raft_consensus.cc:2454] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:43.125880 13867 leader_election.cc:302] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:43.126051 18185 raft_consensus.cc:2781] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.126137 18185 raft_consensus.cc:686] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Becoming Leader. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Running, Role: LEADER
I20240501 19:56:43.126271 18185 consensus_queue.cc:237] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:43.128909 11100 catalog_manager.cc:5484] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921cf2e41f64d66842885512da8de6c (127.5.40.7). New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.145586 11100 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:43.145747 11100 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:43.147898 11100 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 875f9bdd03e74522b95d9585d325bf23
I20240501 19:56:43.148511 13781 tablet_service.cc:1514] Processing DeleteTablet for tablet 875f9bdd03e74522b95d9585d325bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:43 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:43.148568 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 875f9bdd03e74522b95d9585d325bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:43 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:43.148790 18071 tablet_replica.cc:330] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:43.148802 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet 875f9bdd03e74522b95d9585d325bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:43 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:43.148871 18206 tablet_replica.cc:330] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:43.148938 18071 raft_consensus.cc:2229] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:43.149017 18206 raft_consensus.cc:2229] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:43.149076 18071 raft_consensus.cc:2258] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:43.149143 18206 raft_consensus.cc:2258] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:43.149588 18207 tablet_replica.cc:330] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:43.149713 18071 ts_tablet_manager.cc:1875] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:43.149752 18207 raft_consensus.cc:2229] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:43.149786 18206 ts_tablet_manager.cc:1875] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:43.149947 18207 raft_consensus.cc:2258] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:43.150447 18207 ts_tablet_manager.cc:1875] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:43.152040 18071 ts_tablet_manager.cc:1888] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:43.152112 18071 log.cc:1196] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/875f9bdd03e74522b95d9585d325bf23
I20240501 19:56:43.152319 18206 ts_tablet_manager.cc:1888] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:43.152350 18071 ts_tablet_manager.cc:1909] T 875f9bdd03e74522b95d9585d325bf23 P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:43.152385 18206 log.cc:1196] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/875f9bdd03e74522b95d9585d325bf23
I20240501 19:56:43.152635 18206 ts_tablet_manager.cc:1909] T 875f9bdd03e74522b95d9585d325bf23 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:43.152645 18207 ts_tablet_manager.cc:1888] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:43.152720 18207 log.cc:1196] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/875f9bdd03e74522b95d9585d325bf23
I20240501 19:56:43.152940 18207 ts_tablet_manager.cc:1909] T 875f9bdd03e74522b95d9585d325bf23 P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:43.153355 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet 875f9bdd03e74522b95d9585d325bf23 (table rebalancer_dynamic_table_new_name [id=333de63ff3cc4460ab6911a2153bac92]) successfully deleted
I20240501 19:56:43.153331 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 875f9bdd03e74522b95d9585d325bf23 (table rebalancer_dynamic_table_new_name [id=333de63ff3cc4460ab6911a2153bac92]) successfully deleted
I20240501 19:56:43.153620 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet 875f9bdd03e74522b95d9585d325bf23 (table rebalancer_dynamic_table_new_name [id=333de63ff3cc4460ab6911a2153bac92]) successfully deleted
I20240501 19:56:43.246704 18209 raft_consensus.cc:482] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.246847 18209 raft_consensus.cc:504] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.247280 18209 leader_election.cc:288] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:43.247700 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85787888b78240b9b064a8e230c07681" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:43.247701 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85787888b78240b9b064a8e230c07681" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:43.247828 11377 raft_consensus.cc:2454] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:43.247836 11247 raft_consensus.cc:2454] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 0.
I20240501 19:56:43.248134 13605 leader_election.cc:302] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:43.248301 18209 raft_consensus.cc:2781] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.248366 18209 raft_consensus.cc:482] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.248394 18209 raft_consensus.cc:3037] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.249498 18209 raft_consensus.cc:504] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.249855 18209 leader_election.cc:288] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Requested vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:43.249850 11100 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:43.250288 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85787888b78240b9b064a8e230c07681" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:43.250317 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85787888b78240b9b064a8e230c07681" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:43.250401 11247 raft_consensus.cc:3037] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.250463 11377 raft_consensus.cc:3037] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.251242 11247 raft_consensus.cc:2454] T 85787888b78240b9b064a8e230c07681 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:43.251531 11377 raft_consensus.cc:2454] T 85787888b78240b9b064a8e230c07681 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
I20240501 19:56:43.251567 13603 leader_election.cc:302] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:43.251763 18209 raft_consensus.cc:2781] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.251847 18209 raft_consensus.cc:686] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:43.251988 18209 consensus_queue.cc:237] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.255429 11100 catalog_manager.cc:5484] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:43.256315 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 4dbd96c0492b47d187470518466473c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8ff553bfdf694e7e9cbc53d3a24e4c5c]), partition=
I20240501 19:56:43.256364 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 4dbd96c0492b47d187470518466473c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8ff553bfdf694e7e9cbc53d3a24e4c5c]), partition=
I20240501 19:56:43.256567 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dbd96c0492b47d187470518466473c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.256582 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dbd96c0492b47d187470518466473c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.258459 18118 tablet_bootstrap.cc:492] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:43.258786 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 4dbd96c0492b47d187470518466473c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8ff553bfdf694e7e9cbc53d3a24e4c5c]), partition=
I20240501 19:56:43.259045 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dbd96c0492b47d187470518466473c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.259574 18118 tablet_bootstrap.cc:654] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.259763 18211 tablet_bootstrap.cc:492] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:43.260550 18211 tablet_bootstrap.cc:654] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.260665 18118 tablet_bootstrap.cc:492] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:43.260761 18118 ts_tablet_manager.cc:1367] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:43.261198 18118 raft_consensus.cc:348] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:43.261315 18118 raft_consensus.cc:374] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.261346 18118 raft_consensus.cc:729] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.261521 18213 tablet_bootstrap.cc:492] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:43.261468 18118 consensus_queue.cc:260] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:43.261777 18118 ts_tablet_manager.cc:1398] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:43.261983 18211 tablet_bootstrap.cc:492] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:43.262128 18211 ts_tablet_manager.cc:1367] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:43.262305 18213 tablet_bootstrap.cc:654] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.262635 18211 raft_consensus.cc:348] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:43.262773 18211 raft_consensus.cc:374] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.262822 18211 raft_consensus.cc:729] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.263024 18211 consensus_queue.cc:260] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:43.264707 18211 ts_tablet_manager.cc:1398] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:43.266180 18213 tablet_bootstrap.cc:492] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:43.266263 18213 ts_tablet_manager.cc:1367] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:43.266741 18213 raft_consensus.cc:348] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:43.266876 18213 raft_consensus.cc:374] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.266932 18213 raft_consensus.cc:729] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.267060 18213 consensus_queue.cc:260] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:43.267352 18213 ts_tablet_manager.cc:1398] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.325398 17946 raft_consensus.cc:482] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.325521 17946 raft_consensus.cc:504] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:43.325861 17946 leader_election.cc:288] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:43.326155 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbd96c0492b47d187470518466473c7" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:43.326222 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbd96c0492b47d187470518466473c7" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:43.326295 13800 raft_consensus.cc:2454] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:43.326357 14063 raft_consensus.cc:2454] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 0.
I20240501 19:56:43.326599 11580 leader_election.cc:302] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:43.326762 17946 raft_consensus.cc:2781] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.326819 17946 raft_consensus.cc:482] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.326864 17946 raft_consensus.cc:3037] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.327769 17946 raft_consensus.cc:504] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:43.328071 17946 leader_election.cc:288] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:43.328366 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbd96c0492b47d187470518466473c7" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:43.328403 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dbd96c0492b47d187470518466473c7" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:43.328473 13800 raft_consensus.cc:3037] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.328501 14063 raft_consensus.cc:3037] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.329233 14063 raft_consensus.cc:2454] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:43.329236 13800 raft_consensus.cc:2454] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:43.329506 11580 leader_election.cc:302] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:43.329526 18187 raft_consensus.cc:482] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.329615 18187 raft_consensus.cc:504] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.329721 17946 raft_consensus.cc:2781] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.329800 17946 raft_consensus.cc:686] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:43.329921 17946 consensus_queue.cc:237] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:43.330027 18187 leader_election.cc:288] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:43.330265 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200886108633404e917f52c7db2a18d9" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" is_pre_election: true
I20240501 19:56:43.330363 13800 raft_consensus.cc:2454] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:43.330396 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200886108633404e917f52c7db2a18d9" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:43.330507 11511 raft_consensus.cc:2454] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 0.
I20240501 19:56:43.330621 11184 leader_election.cc:302] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:43.330902 18187 raft_consensus.cc:2781] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.330960 18187 raft_consensus.cc:482] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.330996 18187 raft_consensus.cc:3037] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.332068 18187 raft_consensus.cc:504] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.332230 11100 catalog_manager.cc:5484] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.332403 18187 leader_election.cc:288] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:43.332661 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200886108633404e917f52c7db2a18d9" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7"
I20240501 19:56:43.332752 13800 raft_consensus.cc:3037] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.332931 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200886108633404e917f52c7db2a18d9" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:43.333024 11511 raft_consensus.cc:3037] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.333648 13800 raft_consensus.cc:2454] T 200886108633404e917f52c7db2a18d9 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:43.333897 11511 raft_consensus.cc:2454] T 200886108633404e917f52c7db2a18d9 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:43.334200 11181 leader_election.cc:302] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:43.334890 18187 raft_consensus.cc:2781] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.334987 18187 raft_consensus.cc:686] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:43.335119 18187 consensus_queue.cc:237] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.336901 11100 catalog_manager.cc:5484] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 1 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.355404 11247 consensus_queue.cc:237] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:43.356529 13800 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 2 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:43.356887 18223 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:43.358392 18187 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:43.358774 13800 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:43.359975 13930 raft_consensus.cc:1261] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:43.361805 11100 catalog_manager.cc:5484] T c3b35dab7eec4b73b208f687dad46ba2 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 8 to 10, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New cstate: current_term: 2 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:43.364310 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3b35dab7eec4b73b208f687dad46ba2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:43.364512 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b35dab7eec4b73b208f687dad46ba2 with delete_type TABLET_DATA_TOMBSTONED (TS 08711534813d47ba85aaec0771ed5d0f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:43.364753 18071 tablet_replica.cc:330] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:43.364919 18071 raft_consensus.cc:2229] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:43.365026 18071 raft_consensus.cc:2258] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:43.365666 18071 ts_tablet_manager.cc:1875] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:43.367542 18071 ts_tablet_manager.cc:1888] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 19:56:43.367609 18071 log.cc:1196] T c3b35dab7eec4b73b208f687dad46ba2 P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3b35dab7eec4b73b208f687dad46ba2
I20240501 19:56:43.368121 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet c3b35dab7eec4b73b208f687dad46ba2 (table rebalance_test_table_19 [id=033f6222c8e847f9b95939d4969a8b21]) successfully deleted
I20240501 19:56:43.368255 18223 consensus_queue.cc:1035] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:43.369372 13930 raft_consensus.cc:2932] T c3b35dab7eec4b73b208f687dad46ba2 P 6921cf2e41f64d66842885512da8de6c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } } }
I20240501 19:56:43.440480 11100 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:43.444326 14063 raft_consensus.cc:1261] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.444761 18160 consensus_queue.cc:1035] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.445189 13800 raft_consensus.cc:1261] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.445557 18160 consensus_queue.cc:1035] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.447468 11100 catalog_manager.cc:5576] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789) does not have the latest schema for tablet 4dbd96c0492b47d187470518466473c7 (table rebalancer_dynamic_table [id=8ff553bfdf694e7e9cbc53d3a24e4c5c]). Expected version 1 got 0
I20240501 19:56:43.447630 18227 tablet.cc:1722] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:43.447878 18227 tablet.cc:1620] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: MemRowSet was empty: no flush needed.
I20240501 19:56:43.448788 11101 catalog_manager.cc:5576] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041) does not have the latest schema for tablet 4dbd96c0492b47d187470518466473c7 (table rebalancer_dynamic_table [id=8ff553bfdf694e7e9cbc53d3a24e4c5c]). Expected version 1 got 0
I20240501 19:56:43.448836 18035 tablet.cc:1722] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:43.449110 18035 tablet.cc:1620] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:43.450393 11100 catalog_manager.cc:5576] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331) does not have the latest schema for tablet 4dbd96c0492b47d187470518466473c7 (table rebalancer_dynamic_table [id=8ff553bfdf694e7e9cbc53d3a24e4c5c]). Expected version 1 got 0
I20240501 19:56:43.451853 18166 tablet.cc:1722] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:43.452142 18166 tablet.cc:1620] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:43.453685 11100 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=8ff553bfdf694e7e9cbc53d3a24e4c5c] alter complete (version 1)
I20240501 19:56:43.526011 18185 consensus_queue.cc:1035] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.529314 16340 consensus_queue.cc:1035] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.558233 11101 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:43.565768 18227 tablet.cc:1722] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:43.566336 18227 tablet.cc:1620] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: MemRowSet was empty: no flush needed.
I20240501 19:56:43.566593 18035 tablet.cc:1722] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:43.566886 18035 tablet.cc:1620] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:43.567026 18166 tablet.cc:1722] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:43.567276 18166 tablet.cc:1620] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: MemRowSet was empty: no flush needed.
I20240501 19:56:43.573422 11097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=8ff553bfdf694e7e9cbc53d3a24e4c5c] alter complete (version 2)
I20240501 19:56:43.630474 11229 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:43.635236 11491 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:56:43.637111 11624 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:56:43.638062 13781 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:43.648150 13648 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:43.660686 13910 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:43.662717 14043 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:43.672544 11359 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:43.677598 11097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:43.685223 18227 tablet.cc:1722] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:43.685796 18035 tablet.cc:1722] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:43.688885 18166 tablet.cc:1722] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:43.691130 11102 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=8ff553bfdf694e7e9cbc53d3a24e4c5c] alter complete (version 3)
I20240501 19:56:43.742491 18208 consensus_queue.cc:1035] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.742584 13800 consensus_queue.cc:237] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:43.744196 14063 raft_consensus.cc:1261] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.744264 11249 raft_consensus.cc:1261] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 96fc18dbb5fd41afb9bebf805e8d33a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.745615 18302 consensus_queue.cc:1035] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:43.747555 13734 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 -> Peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): Couldn't send request to peer 08711534813d47ba85aaec0771ed5d0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0da88b92a3fc4f1a89e7eb990557d7d7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.748510 18301 consensus_queue.cc:1035] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.748510 18209 consensus_queue.cc:1035] T 85787888b78240b9b064a8e230c07681 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.750356 11247 raft_consensus.cc:2932] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:43.750753 18302 raft_consensus.cc:2932] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:43.752229 14063 raft_consensus.cc:2932] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:43.752985 11106 catalog_manager.cc:5484] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:43.760936 18223 consensus_queue.cc:1035] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.764780 18187 consensus_queue.cc:1035] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.765272 11643 consensus_queue.cc:237] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:43.766806 13930 raft_consensus.cc:1261] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.766844 13799 raft_consensus.cc:1261] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:43.767086 11579 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): Couldn't send request to peer ef60ab93b49c4346af4f62d3b98558b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6caecf09f4114855bddc470eb96bd5af. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.767406 17946 consensus_queue.cc:1035] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.767611 18126 consensus_queue.cc:1035] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.769758 17946 raft_consensus.cc:2932] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:43.771638 13800 raft_consensus.cc:2932] T 6caecf09f4114855bddc470eb96bd5af P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:43.772213 13930 raft_consensus.cc:2932] T 6caecf09f4114855bddc470eb96bd5af P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:43.773329 11097 catalog_manager.cc:5484] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.781700 11644 consensus_queue.cc:237] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:43.783088 14063 raft_consensus.cc:1261] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.783306 13669 raft_consensus.cc:1261] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.783514 18320 consensus_queue.cc:1035] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:43.783704 11577 consensus_peers.cc:485] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 -> Peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): Couldn't send request to peer 10ee60ead7b94c93ab749ec16a04b0dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e16610fde0b47a199efc533627ae0ab. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.783885 18320 consensus_queue.cc:1035] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.785210 18126 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:43.786800 14063 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:43.787106 13669 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:43.789033 11102 catalog_manager.cc:5484] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.794412 11097 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:43.794561 11097 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:43.796612 11097 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 4dbd96c0492b47d187470518466473c7
I20240501 19:56:43.797125 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dbd96c0492b47d187470518466473c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:43 UTC) from {username='slave'} at 127.0.0.1:35718
I20240501 19:56:43.797237 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dbd96c0492b47d187470518466473c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:43 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:43.797577 18335 tablet_replica.cc:330] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:43.797566 13781 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dbd96c0492b47d187470518466473c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:43 UTC) from {username='slave'} at 127.0.0.1:53834
I20240501 19:56:43.797703 18335 raft_consensus.cc:2229] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:43.797920 18336 tablet_replica.cc:330] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: stopping tablet replica
I20240501 19:56:43.797978 14063 consensus_queue.cc:237] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:43.798116 18334 tablet_replica.cc:330] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: stopping tablet replica
I20240501 19:56:43.798164 18336 raft_consensus.cc:2229] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:43.798257 18334 raft_consensus.cc:2229] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:43.798317 18336 raft_consensus.cc:2258] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:43.798049 18335 raft_consensus.cc:2258] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:43.798454 18334 raft_consensus.cc:2258] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:43.798980 18335 ts_tablet_manager.cc:1875] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:43.798987 18334 ts_tablet_manager.cc:1875] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:43.799093 18336 ts_tablet_manager.cc:1875] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:43.799989 11377 raft_consensus.cc:1261] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:43.800247 13997 consensus_peers.cc:485] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a -> Peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): Couldn't send request to peer 8ad18cd54c6340fd92b874157875fa49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bc4ce342be94b309a35450d3c59a6ce. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.800570 18228 consensus_queue.cc:1035] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.801625 18335 ts_tablet_manager.cc:1888] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:43.801730 18334 ts_tablet_manager.cc:1888] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:43.801743 18335 log.cc:1196] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/4dbd96c0492b47d187470518466473c7
I20240501 19:56:43.801867 18334 log.cc:1196] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-3/wal/wals/4dbd96c0492b47d187470518466473c7
I20240501 19:56:43.802017 18335 ts_tablet_manager.cc:1909] T 4dbd96c0492b47d187470518466473c7 P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:43.802336 13800 raft_consensus.cc:1261] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.802363 18334 ts_tablet_manager.cc:1909] T 4dbd96c0492b47d187470518466473c7 P 8ad18cd54c6340fd92b874157875fa49: Deleting consensus metadata
I20240501 19:56:43.802742 18228 consensus_queue.cc:1035] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.802817 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet 4dbd96c0492b47d187470518466473c7 (table rebalancer_dynamic_table_new_name [id=8ff553bfdf694e7e9cbc53d3a24e4c5c]) successfully deleted
I20240501 19:56:43.802996 11090 catalog_manager.cc:4830] TS 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): tablet 4dbd96c0492b47d187470518466473c7 (table rebalancer_dynamic_table_new_name [id=8ff553bfdf694e7e9cbc53d3a24e4c5c]) successfully deleted
I20240501 19:56:43.803988 18336 ts_tablet_manager.cc:1888] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:43.804111 18336 log.cc:1196] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-5/wal/wals/4dbd96c0492b47d187470518466473c7
I20240501 19:56:43.804379 18336 ts_tablet_manager.cc:1909] T 4dbd96c0492b47d187470518466473c7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Deleting consensus metadata
I20240501 19:56:43.804246 18228 raft_consensus.cc:2932] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:43.804988 11092 catalog_manager.cc:4830] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): tablet 4dbd96c0492b47d187470518466473c7 (table rebalancer_dynamic_table_new_name [id=8ff553bfdf694e7e9cbc53d3a24e4c5c]) successfully deleted
I20240501 19:56:43.805835 13800 raft_consensus.cc:2932] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:43.807830 11377 raft_consensus.cc:2932] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:43.808284 11097 catalog_manager.cc:5484] T 0bc4ce342be94b309a35450d3c59a6ce P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:43.816181 11377 consensus_queue.cc:237] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } }
I20240501 19:56:43.817545 13799 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:43.817559 11511 raft_consensus.cc:1261] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:43.817613 11313 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6c1d3814e364e718449c3e4379b31ed. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.817917 18347 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:43.818130 16220 consensus_queue.cc:1035] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:43.819926 18307 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:43.820360 11511 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:43.820461 13799 raft_consensus.cc:2932] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:43.823046 11102 catalog_manager.cc:5484] T e6c1d3814e364e718449c3e4379b31ed P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 6 to 7, NON_VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) added. New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: false } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: NON_VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: true } } }
I20240501 19:56:43.830654 13930 consensus_queue.cc:237] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:43.831930 11377 raft_consensus.cc:1261] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:43.831939 13866 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c -> Peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041): Couldn't send request to peer 8ad18cd54c6340fd92b874157875fa49. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b85f5acd65914c2d8f1cc48d668c40eb. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.832298 18355 consensus_queue.cc:1035] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.832417 13669 raft_consensus.cc:1261] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.832763 18355 consensus_queue.cc:1035] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.833722 16340 raft_consensus.cc:2932] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:43.834165 11377 raft_consensus.cc:2932] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:43.834899 13669 raft_consensus.cc:2932] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:43.836851 11106 catalog_manager.cc:5484] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index -1 to 2, NON_VOTER 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) added. New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } } }
I20240501 19:56:43.856058 13669 consensus_queue.cc:237] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:43.857511 14063 raft_consensus.cc:1261] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.857784 13930 raft_consensus.cc:1261] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.857903 18362 consensus_queue.cc:1035] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:43.858263 13603 consensus_peers.cc:485] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965): Couldn't send request to peer 10ee60ead7b94c93ab749ec16a04b0dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee5178fb00d741d49d42168a71e5d23a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.858439 18362 consensus_queue.cc:1035] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.859935 18208 raft_consensus.cc:2932] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:43.860502 14063 raft_consensus.cc:2932] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:43.862893 11102 catalog_manager.cc:5484] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.863479 13930 raft_consensus.cc:2932] T ee5178fb00d741d49d42168a71e5d23a P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } } }
I20240501 19:56:43.874507 11511 consensus_queue.cc:237] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:43.876158 11377 raft_consensus.cc:1261] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.876293 14063 raft_consensus.cc:1261] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:43.876466 11448 consensus_peers.cc:485] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a5f03eecaec4b929723c8f02a185606. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.876627 16302 consensus_queue.cc:1035] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.876801 18371 consensus_queue.cc:1035] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.878459 18371 raft_consensus.cc:2932] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:43.878832 14063 raft_consensus.cc:2932] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:43.879045 11377 raft_consensus.cc:2932] T 6a5f03eecaec4b929723c8f02a185606 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:43.881990 11106 catalog_manager.cc:5484] T 6a5f03eecaec4b929723c8f02a185606 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index -1 to 2, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:43.889837 14063 consensus_queue.cc:237] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:43.891115 13799 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:43.891146 11644 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:43.891414 13996 consensus_peers.cc:485] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): Couldn't send request to peer 08711534813d47ba85aaec0771ed5d0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66bdddd3550b4ae8a97535e4747a5234. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.891554 16195 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:43.891742 18340 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:43.893393 18228 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:43.893859 11644 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:43.893944 13799 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:43.896533 11102 catalog_manager.cc:5484] T 66bdddd3550b4ae8a97535e4747a5234 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: config changed from index 4 to 5, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:43.898478 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:43.903549 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 05c47f76981c418aad6e12afc32501ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3b03f2fb37964559b6161ed99fb86d81]), partition=
I20240501 19:56:43.903553 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 05c47f76981c418aad6e12afc32501ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3b03f2fb37964559b6161ed99fb86d81]), partition=
I20240501 19:56:43.903632 11491 tablet_service.cc:1467] Processing CreateTablet for tablet 05c47f76981c418aad6e12afc32501ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3b03f2fb37964559b6161ed99fb86d81]), partition=
I20240501 19:56:43.903789 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c47f76981c418aad6e12afc32501ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.903805 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c47f76981c418aad6e12afc32501ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.903818 11491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c47f76981c418aad6e12afc32501ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.906026 18387 tablet_bootstrap.cc:492] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:43.906297 18389 tablet_bootstrap.cc:492] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:43.907037 18387 tablet_bootstrap.cc:654] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.907629 18388 tablet_bootstrap.cc:492] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:43.908102 11511 consensus_queue.cc:237] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:43.908463 18387 tablet_bootstrap.cc:492] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:43.908509 18388 tablet_bootstrap.cc:654] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.908589 18387 ts_tablet_manager.cc:1367] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:43.909065 18387 raft_consensus.cc:348] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.909744 18387 raft_consensus.cc:374] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.909785 13930 raft_consensus.cc:1261] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.909814 18387 raft_consensus.cc:729] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.910063 18389 tablet_bootstrap.cc:654] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.910254 18257 consensus_queue.cc:1035] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.910310 13799 raft_consensus.cc:1261] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.910642 18257 consensus_queue.cc:1035] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.911202 18389 tablet_bootstrap.cc:492] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:43.911331 18389 ts_tablet_manager.cc:1367] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
W20240501 19:56:43.911715 11447 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): Couldn't send request to peer ef60ab93b49c4346af4f62d3b98558b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33be1f7276c74d49b42c8761e3e4cd4e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.911938 18389 raft_consensus.cc:348] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.910018 18387 consensus_queue.cc:260] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.912168 18388 tablet_bootstrap.cc:492] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: No bootstrap required, opened a new log
I20240501 19:56:43.912313 18389 raft_consensus.cc:374] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.912385 18389 raft_consensus.cc:729] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.912391 18371 raft_consensus.cc:2932] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:43.912426 11099 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:43.912735 18389 consensus_queue.cc:260] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.913054 18387 ts_tablet_manager.cc:1398] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:43.914448 13930 raft_consensus.cc:2932] T 33be1f7276c74d49b42c8761e3e4cd4e P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:43.914934 18389 ts_tablet_manager.cc:1398] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:43.917632 11106 catalog_manager.cc:5484] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.919622 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 8efaba1013424acd949123503dc503b8 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=0af31289105a442fbbdece50e8d921b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.919575 13799 raft_consensus.cc:2932] T 33be1f7276c74d49b42c8761e3e4cd4e P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:43.919927 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efaba1013424acd949123503dc503b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.921058 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 8efaba1013424acd949123503dc503b8 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=0af31289105a442fbbdece50e8d921b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.921330 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efaba1013424acd949123503dc503b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.921557 13648 tablet_service.cc:1467] Processing CreateTablet for tablet 8efaba1013424acd949123503dc503b8 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=0af31289105a442fbbdece50e8d921b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.921587 14042 tablet_service.cc:1467] Processing CreateTablet for tablet 2f09627997b5410a9d4f8149cf0f01b4 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=0af31289105a442fbbdece50e8d921b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.921772 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f09627997b5410a9d4f8149cf0f01b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.921826 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efaba1013424acd949123503dc503b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.922438 13649 tablet_service.cc:1467] Processing CreateTablet for tablet 2f09627997b5410a9d4f8149cf0f01b4 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=0af31289105a442fbbdece50e8d921b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.922648 13649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f09627997b5410a9d4f8149cf0f01b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.923539 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4cf91c71044cf6a37528e8c23954c0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=0af31289105a442fbbdece50e8d921b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.923758 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4cf91c71044cf6a37528e8c23954c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.924186 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4cf91c71044cf6a37528e8c23954c0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=0af31289105a442fbbdece50e8d921b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.924427 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4cf91c71044cf6a37528e8c23954c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.926273 11624 tablet_service.cc:1467] Processing CreateTablet for tablet 2f09627997b5410a9d4f8149cf0f01b4 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=0af31289105a442fbbdece50e8d921b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.926570 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f09627997b5410a9d4f8149cf0f01b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.926582 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4cf91c71044cf6a37528e8c23954c0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=0af31289105a442fbbdece50e8d921b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.926785 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4cf91c71044cf6a37528e8c23954c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.929141 18355 raft_consensus.cc:482] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.929227 18355 raft_consensus.cc:504] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.918362 18388 ts_tablet_manager.cc:1367] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:43.929623 18355 leader_election.cc:288] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:43.929867 18388 raft_consensus.cc:348] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.929982 18388 raft_consensus.cc:374] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.930012 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c47f76981c418aad6e12afc32501ed" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:43.930013 18388 raft_consensus.cc:729] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.930209 18388 consensus_queue.cc:260] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.930399 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c47f76981c418aad6e12afc32501ed" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:43.930510 14063 raft_consensus.cc:2454] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:43.930656 11511 raft_consensus.cc:2454] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 0.
I20240501 19:56:43.930833 18387 tablet_bootstrap.cc:492] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c: Bootstrap starting.
I20240501 19:56:43.930979 13865 leader_election.cc:302] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:43.931115 18388 ts_tablet_manager.cc:1398] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:43.931352 18355 raft_consensus.cc:2781] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.931417 18355 raft_consensus.cc:482] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.931481 18355 raft_consensus.cc:3037] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.931618 18396 tablet_bootstrap.cc:492] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:43.931677 18389 tablet_bootstrap.cc:492] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:43.932399 18355 raft_consensus.cc:504] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.932713 18355 leader_election.cc:288] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:43.933223 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c47f76981c418aad6e12afc32501ed" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:43.933264 18389 tablet_bootstrap.cc:654] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.933322 11511 raft_consensus.cc:3037] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.933467 18396 tablet_bootstrap.cc:654] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.934029 18402 tablet_bootstrap.cc:492] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:43.934247 11511 raft_consensus.cc:2454] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:43.934523 13865 leader_election.cc:302] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 6921cf2e41f64d66842885512da8de6c; no voters:
I20240501 19:56:43.934676 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c47f76981c418aad6e12afc32501ed" candidate_uuid: "6921cf2e41f64d66842885512da8de6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:43.934803 14063 raft_consensus.cc:3037] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.935103 18397 tablet_bootstrap.cc:492] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:43.935261 18387 tablet_bootstrap.cc:654] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.935734 14063 raft_consensus.cc:2454] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6921cf2e41f64d66842885512da8de6c in term 1.
I20240501 19:56:43.936007 18355 raft_consensus.cc:2781] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.936057 18397 tablet_bootstrap.cc:654] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.936086 18355 raft_consensus.cc:686] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [term 1 LEADER]: Becoming Leader. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Running, Role: LEADER
I20240501 19:56:43.936210 18355 consensus_queue.cc:237] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } }
I20240501 19:56:43.936875 18387 tablet_bootstrap.cc:492] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c: No bootstrap required, opened a new log
I20240501 19:56:43.936951 18387 ts_tablet_manager.cc:1367] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 19:56:43.937152 18402 tablet_bootstrap.cc:654] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.937436 18387 raft_consensus.cc:348] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:43.937544 18387 raft_consensus.cc:374] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.937574 18387 raft_consensus.cc:729] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921cf2e41f64d66842885512da8de6c, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.937709 18389 tablet_bootstrap.cc:492] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:43.937693 18387 consensus_queue.cc:260] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:43.937774 18389 ts_tablet_manager.cc:1367] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:43.937959 18387 ts_tablet_manager.cc:1398] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:43.938300 18389 raft_consensus.cc:348] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.938409 18389 raft_consensus.cc:374] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.938441 18389 raft_consensus.cc:729] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.938553 18389 consensus_queue.cc:260] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.938737 18397 tablet_bootstrap.cc:492] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:43.938812 18397 ts_tablet_manager.cc:1367] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:43.938827 18389 ts_tablet_manager.cc:1398] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.939061 18389 tablet_bootstrap.cc:492] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:43.939502 18397 raft_consensus.cc:348] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.939668 18397 raft_consensus.cc:374] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.939738 18397 raft_consensus.cc:729] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.939930 18397 consensus_queue.cc:260] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.940050 18389 tablet_bootstrap.cc:654] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.940287 18397 ts_tablet_manager.cc:1398] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.940320 11099 catalog_manager.cc:5484] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6921cf2e41f64d66842885512da8de6c (127.5.40.7). New cstate: current_term: 1 leader_uuid: "6921cf2e41f64d66842885512da8de6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.941287 18397 tablet_bootstrap.cc:492] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:43.942261 18397 tablet_bootstrap.cc:654] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.944757 18402 tablet_bootstrap.cc:492] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49: No bootstrap required, opened a new log
I20240501 19:56:43.944908 18402 ts_tablet_manager.cc:1367] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:43.945283 18397 tablet_bootstrap.cc:492] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:43.945461 18402 raft_consensus.cc:348] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.945580 18402 raft_consensus.cc:374] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.945608 18402 raft_consensus.cc:729] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.945719 18402 consensus_queue.cc:260] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.946640 18397 ts_tablet_manager.cc:1367] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:43.947077 18397 raft_consensus.cc:348] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.947172 18397 raft_consensus.cc:374] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.947202 18397 raft_consensus.cc:729] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.947731 18402 ts_tablet_manager.cc:1398] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:43.947337 18397 consensus_queue.cc:260] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.948961 18397 ts_tablet_manager.cc:1398] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:43.950608 18396 tablet_bootstrap.cc:492] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:43.950711 18396 ts_tablet_manager.cc:1367] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 19:56:43.950955 18397 tablet_bootstrap.cc:492] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:43.951180 18396 raft_consensus.cc:348] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.951297 18396 raft_consensus.cc:374] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.951346 18396 raft_consensus.cc:729] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.951522 18396 consensus_queue.cc:260] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.952544 18397 tablet_bootstrap.cc:654] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.952500 11511 consensus_queue.cc:237] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } }
I20240501 19:56:43.953891 11644 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:43.953969 11448 consensus_peers.cc:485] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fc7da07b6c5414bb305264962d0f218. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.954216 18396 ts_tablet_manager.cc:1398] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:43.954236 18371 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:43.954371 18396 tablet_bootstrap.cc:492] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:43.955302 18396 tablet_bootstrap.cc:654] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.956629 18389 tablet_bootstrap.cc:492] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:43.956699 18396 tablet_bootstrap.cc:492] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7: No bootstrap required, opened a new log
I20240501 19:56:43.956779 18389 ts_tablet_manager.cc:1367] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 19:56:43.956825 18396 ts_tablet_manager.cc:1367] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:43.957233 18396 raft_consensus.cc:348] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:43.957332 18396 raft_consensus.cc:374] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.957358 18396 raft_consensus.cc:729] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.957492 18396 consensus_queue.cc:260] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:43.957412 18389 raft_consensus.cc:348] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.957578 18389 raft_consensus.cc:374] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.957634 18389 raft_consensus.cc:729] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.957805 18396 ts_tablet_manager.cc:1398] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:43.957788 18389 consensus_queue.cc:260] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:43.958344 14063 raft_consensus.cc:1261] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:43.958515 18397 tablet_bootstrap.cc:492] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:43.958588 18397 ts_tablet_manager.cc:1367] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 19:56:43.958803 18371 consensus_queue.cc:1035] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:43.958993 18397 raft_consensus.cc:348] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:43.959136 18397 raft_consensus.cc:374] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.959204 18397 raft_consensus.cc:729] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.959353 18397 consensus_queue.cc:260] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:43.959730 18397 ts_tablet_manager.cc:1398] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.960929 18389 ts_tablet_manager.cc:1398] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:43.961045 18371 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:43.962986 11644 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:43.963888 11097 catalog_manager.cc:5484] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New cstate: current_term: 1 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.973479 14063 raft_consensus.cc:2932] T 1fc7da07b6c5414bb305264962d0f218 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { promote: false } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: NON_VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { promote: true } } }
I20240501 19:56:43.980787 13669 consensus_queue.cc:237] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:43.982201 13799 raft_consensus.cc:1261] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.982532 14063 raft_consensus.cc:1261] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:43.982723 13605 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185): Couldn't send request to peer ef60ab93b49c4346af4f62d3b98558b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc5eb3e030774199836cff2a657e8d56. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.982841 18362 consensus_queue.cc:1035] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.983042 18208 consensus_queue.cc:1035] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:43.985039 18208 raft_consensus.cc:2932] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:43.986836 14063 raft_consensus.cc:2932] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:43.988121 13799 raft_consensus.cc:2932] T fc5eb3e030774199836cff2a657e8d56 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } } }
I20240501 19:56:43.988292 11106 catalog_manager.cc:5484] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index -1 to 2, NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) added. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.996690 11644 consensus_queue.cc:237] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:43.998040 14063 raft_consensus.cc:1261] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.998225 13930 raft_consensus.cc:1261] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.998566 18126 consensus_queue.cc:1035] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:43.998800 11577 consensus_peers.cc:485] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): Couldn't send request to peer 08711534813d47ba85aaec0771ed5d0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9dad5cfa9b7a4a808eb9d0bb81f2a2c6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:43.999011 18126 consensus_queue.cc:1035] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:44.000432 18320 raft_consensus.cc:2932] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:44.002055 13930 raft_consensus.cc:2932] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:44.002565 14063 raft_consensus.cc:2932] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:44.003003 11099 catalog_manager.cc:5484] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:44.012318 13669 consensus_queue.cc:237] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:44.013856 13799 raft_consensus.cc:1261] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.013893 13930 raft_consensus.cc:1261] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:44.014163 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): Couldn't send request to peer 08711534813d47ba85aaec0771ed5d0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19d0d6aaa47e4f7eaa80ec5e0207d51d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.014292 18425 consensus_queue.cc:1035] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:44.014504 18426 consensus_queue.cc:1035] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:44.016261 18426 raft_consensus.cc:2932] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:44.016964 13930 raft_consensus.cc:2932] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:44.018776 13799 raft_consensus.cc:2932] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:44.020637 11102 catalog_manager.cc:5484] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 6921cf2e41f64d66842885512da8de6c reported cstate change: config changed from index -1 to 2, NON_VOTER 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) added. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } } }
I20240501 19:56:44.034989 18302 raft_consensus.cc:482] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.035107 18302 raft_consensus.cc:504] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:44.035573 18302 leader_election.cc:288] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:44.035995 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4cf91c71044cf6a37528e8c23954c0" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:44.035987 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4cf91c71044cf6a37528e8c23954c0" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c" is_pre_election: true
I20240501 19:56:44.036114 13930 raft_consensus.cc:2454] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:44.036118 13669 raft_consensus.cc:2454] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:44.036386 13734 leader_election.cc:302] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:44.036671 18302 raft_consensus.cc:2781] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.036736 18302 raft_consensus.cc:482] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.036775 18302 raft_consensus.cc:3037] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.037853 18302 raft_consensus.cc:504] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:44.038213 18302 leader_election.cc:288] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:44.039047 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4cf91c71044cf6a37528e8c23954c0" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6921cf2e41f64d66842885512da8de6c"
I20240501 19:56:44.039156 13930 raft_consensus.cc:3037] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.039151 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4cf91c71044cf6a37528e8c23954c0" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:44.039284 13669 raft_consensus.cc:3037] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.040272 13669 raft_consensus.cc:2454] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:44.040688 13930 raft_consensus.cc:2454] T 2d4cf91c71044cf6a37528e8c23954c0 P 6921cf2e41f64d66842885512da8de6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:44.041220 13734 leader_election.cc:302] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:44.041479 18302 raft_consensus.cc:2781] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.041565 18302 raft_consensus.cc:686] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:44.041685 18302 consensus_queue.cc:237] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:44.044180 11097 catalog_manager.cc:5484] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:44.045109 11106 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 19:56:44.049496 11511 raft_consensus.cc:1261] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.049614 14063 raft_consensus.cc:1261] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 6921cf2e41f64d66842885512da8de6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.049969 18355 consensus_queue.cc:1035] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.050248 18355 consensus_queue.cc:1035] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.052834 11106 catalog_manager.cc:5576] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965) does not have the latest schema for tablet 05c47f76981c418aad6e12afc32501ed (table rebalancer_dynamic_table [id=3b03f2fb37964559b6161ed99fb86d81]). Expected version 1 got 0
I20240501 19:56:44.053474 11097 catalog_manager.cc:5576] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789) does not have the latest schema for tablet 05c47f76981c418aad6e12afc32501ed (table rebalancer_dynamic_table [id=3b03f2fb37964559b6161ed99fb86d81]). Expected version 1 got 0
I20240501 19:56:44.054777 18467 tablet.cc:1722] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:44.055088 18467 tablet.cc:1620] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:44.056941 18035 tablet.cc:1722] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:44.057232 18035 tablet.cc:1620] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:44.059140 11106 catalog_manager.cc:5576] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965) does not have the latest schema for tablet 05c47f76981c418aad6e12afc32501ed (table rebalancer_dynamic_table [id=3b03f2fb37964559b6161ed99fb86d81]). Expected version 1 got 0
I20240501 19:56:44.059300 18468 tablet.cc:1722] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:56:44.059645 18468 tablet.cc:1620] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:44.061547 11097 catalog_manager.cc:5576] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789) does not have the latest schema for tablet 05c47f76981c418aad6e12afc32501ed (table rebalancer_dynamic_table [id=3b03f2fb37964559b6161ed99fb86d81]). Expected version 1 got 0
I20240501 19:56:44.061774 11102 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=3b03f2fb37964559b6161ed99fb86d81] alter complete (version 1)
I20240501 19:56:44.087357 18302 raft_consensus.cc:482] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.087491 18302 raft_consensus.cc:504] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:44.087921 18302 leader_election.cc:288] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:44.088202 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efaba1013424acd949123503dc503b8" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:44.088258 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efaba1013424acd949123503dc503b8" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:44.088354 14063 raft_consensus.cc:2454] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:44.088373 13669 raft_consensus.cc:2454] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 0.
I20240501 19:56:44.088655 13734 leader_election.cc:302] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:44.088948 18302 raft_consensus.cc:2781] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.089010 18302 raft_consensus.cc:482] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.089043 18302 raft_consensus.cc:3037] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.090091 18302 raft_consensus.cc:504] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:44.090374 18302 leader_election.cc:288] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:44.090829 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efaba1013424acd949123503dc503b8" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:44.090824 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efaba1013424acd949123503dc503b8" candidate_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:44.090945 13669 raft_consensus.cc:3037] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.090945 14063 raft_consensus.cc:3037] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.091866 13669 raft_consensus.cc:2454] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:44.091939 14063 raft_consensus.cc:2454] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc18dbb5fd41afb9bebf805e8d33a7 in term 1.
I20240501 19:56:44.092185 13734 leader_election.cc:302] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, 96fc18dbb5fd41afb9bebf805e8d33a7; no voters:
I20240501 19:56:44.092466 18302 raft_consensus.cc:2781] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.092564 18302 raft_consensus.cc:686] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc18dbb5fd41afb9bebf805e8d33a7, State: Running, Role: LEADER
I20240501 19:56:44.092702 18302 consensus_queue.cc:237] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:44.094987 11097 catalog_manager.cc:5484] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6). New cstate: current_term: 1 leader_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:44.164346 11097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 19:56:44.169410 18467 tablet.cc:1722] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:44.169760 18467 tablet.cc:1620] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: MemRowSet was empty: no flush needed.
I20240501 19:56:44.170411 18035 tablet.cc:1722] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:44.170751 18035 tablet.cc:1620] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: MemRowSet was empty: no flush needed.
I20240501 19:56:44.171744 18468 tablet.cc:1722] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 19:56:44.172062 18468 tablet.cc:1620] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: MemRowSet was empty: no flush needed.
I20240501 19:56:44.175789 11097 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table [id=3b03f2fb37964559b6161ed99fb86d81] alter complete (version 2)
I20240501 19:56:44.185770 16195 raft_consensus.cc:482] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.185881 16195 raft_consensus.cc:504] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:44.186259 16195 leader_election.cc:288] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:44.186587 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f09627997b5410a9d4f8149cf0f01b4" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
I20240501 19:56:44.186599 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f09627997b5410a9d4f8149cf0f01b4" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:44.186719 13669 raft_consensus.cc:2454] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:44.186718 11644 raft_consensus.cc:2454] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:44.187023 13996 leader_election.cc:302] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
I20240501 19:56:44.187186 16195 raft_consensus.cc:2781] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.187235 16195 raft_consensus.cc:482] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.187258 16195 raft_consensus.cc:3037] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.188303 16195 raft_consensus.cc:504] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:44.188632 16195 leader_election.cc:288] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:44.188985 11644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f09627997b5410a9d4f8149cf0f01b4" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49"
I20240501 19:56:44.188987 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f09627997b5410a9d4f8149cf0f01b4" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:44.189096 11644 raft_consensus.cc:3037] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.189119 13669 raft_consensus.cc:3037] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.190053 11644 raft_consensus.cc:2454] T 2f09627997b5410a9d4f8149cf0f01b4 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:44.190124 13669 raft_consensus.cc:2454] T 2f09627997b5410a9d4f8149cf0f01b4 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:44.190371 13997 leader_election.cc:302] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8ad18cd54c6340fd92b874157875fa49; no voters:
I20240501 19:56:44.190526 16195 raft_consensus.cc:2781] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.190601 16195 raft_consensus.cc:686] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:44.190726 16195 consensus_queue.cc:237] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:44.192839 11097 catalog_manager.cc:5484] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:44.223472 18522 ts_tablet_manager.cc:902] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:44.223994 18522 tablet_copy_client.cc:308] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:44.224459 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet 6caecf09f4114855bddc470eb96bd5af from peer ef60ab93b49c4346af4f62d3b98558b8 ({username='slave'} at 127.5.40.2:50491)
I20240501 19:56:44.224558 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet 6caecf09f4114855bddc470eb96bd5af from peer ef60ab93b49c4346af4f62d3b98558b8 at {username='slave'} at 127.5.40.2:50491: session id = ef60ab93b49c4346af4f62d3b98558b8-6caecf09f4114855bddc470eb96bd5af
I20240501 19:56:44.225303 11664 tablet_copy_source_session.cc:205] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.226153 18522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caecf09f4114855bddc470eb96bd5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.228408 18522 tablet_copy_client.cc:783] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.228550 18522 tablet_copy_client.cc:647] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:44.229562 18522 tablet_copy_client.cc:523] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:44.230662 18522 tablet_bootstrap.cc:492] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:44.232436 18527 ts_tablet_manager.cc:902] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:44.232867 18528 ts_tablet_manager.cc:902] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: Initiating tablet copy from peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:44.233028 18527 tablet_copy_client.cc:308] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
I20240501 19:56:44.233383 18528 tablet_copy_client.cc:308] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.6:46331
I20240501 19:56:44.233542 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet 3e16610fde0b47a199efc533627ae0ab from peer 10ee60ead7b94c93ab749ec16a04b0dc ({username='slave'} at 127.5.40.1:49437)
I20240501 19:56:44.233647 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet 3e16610fde0b47a199efc533627ae0ab from peer 10ee60ead7b94c93ab749ec16a04b0dc at {username='slave'} at 127.5.40.1:49437: session id = 10ee60ead7b94c93ab749ec16a04b0dc-3e16610fde0b47a199efc533627ae0ab
I20240501 19:56:44.233800 13821 tablet_copy_service.cc:140] P 96fc18dbb5fd41afb9bebf805e8d33a7: Received BeginTabletCopySession request for tablet 0da88b92a3fc4f1a89e7eb990557d7d7 from peer 08711534813d47ba85aaec0771ed5d0f ({username='slave'} at 127.5.40.3:60827)
I20240501 19:56:44.233888 13821 tablet_copy_service.cc:161] P 96fc18dbb5fd41afb9bebf805e8d33a7: Beginning new tablet copy session on tablet 0da88b92a3fc4f1a89e7eb990557d7d7 from peer 08711534813d47ba85aaec0771ed5d0f at {username='slave'} at 127.5.40.3:60827: session id = 08711534813d47ba85aaec0771ed5d0f-0da88b92a3fc4f1a89e7eb990557d7d7
I20240501 19:56:44.234608 11664 tablet_copy_source_session.cc:205] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.234755 13821 tablet_copy_source_session.cc:205] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 96fc18dbb5fd41afb9bebf805e8d33a7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.235015 18522 tablet_bootstrap.cc:492] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.235219 18522 tablet_bootstrap.cc:492] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap complete.
I20240501 19:56:44.235373 18522 ts_tablet_manager.cc:1367] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 19:56:44.235828 18522 raft_consensus.cc:348] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:44.235952 18522 raft_consensus.cc:729] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: LEARNER
I20240501 19:56:44.236045 18522 consensus_queue.cc:260] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:44.236326 18522 ts_tablet_manager.cc:1398] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.236596 18527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e16610fde0b47a199efc533627ae0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.237277 18528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da88b92a3fc4f1a89e7eb990557d7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.238206 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session ef60ab93b49c4346af4f62d3b98558b8-6caecf09f4114855bddc470eb96bd5af received from {username='slave'} at 127.5.40.2:50491
I20240501 19:56:44.238272 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session ef60ab93b49c4346af4f62d3b98558b8-6caecf09f4114855bddc470eb96bd5af on tablet 6caecf09f4114855bddc470eb96bd5af with peer ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:44.238925 18527 tablet_copy_client.cc:783] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.239063 18527 tablet_copy_client.cc:647] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:44.240851 18527 tablet_copy_client.cc:523] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:44.241827 18528 tablet_copy_client.cc:783] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.241977 18528 tablet_copy_client.cc:647] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:44.242265 18527 tablet_bootstrap.cc:492] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:44.242967 18528 tablet_copy_client.cc:523] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:44.244222 18528 tablet_bootstrap.cc:492] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:44.247563 18528 tablet_bootstrap.cc:492] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.247790 18528 tablet_bootstrap.cc:492] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap complete.
I20240501 19:56:44.247982 18528 ts_tablet_manager.cc:1367] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:44.248385 18528 raft_consensus.cc:348] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:44.248513 18528 raft_consensus.cc:729] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: LEARNER
I20240501 19:56:44.248602 18528 consensus_queue.cc:260] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:44.248646 18527 tablet_bootstrap.cc:492] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.248867 18528 ts_tablet_manager.cc:1398] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:44.248881 18527 tablet_bootstrap.cc:492] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap complete.
I20240501 19:56:44.249025 18527 ts_tablet_manager.cc:1367] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240501 19:56:44.249327 13821 tablet_copy_service.cc:342] P 96fc18dbb5fd41afb9bebf805e8d33a7: Request end of tablet copy session 08711534813d47ba85aaec0771ed5d0f-0da88b92a3fc4f1a89e7eb990557d7d7 received from {username='slave'} at 127.5.40.3:60827
I20240501 19:56:44.249397 13821 tablet_copy_service.cc:433] P 96fc18dbb5fd41afb9bebf805e8d33a7: ending tablet copy session 08711534813d47ba85aaec0771ed5d0f-0da88b92a3fc4f1a89e7eb990557d7d7 on tablet 0da88b92a3fc4f1a89e7eb990557d7d7 with peer 08711534813d47ba85aaec0771ed5d0f
I20240501 19:56:44.249435 18527 raft_consensus.cc:348] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:44.249541 18527 raft_consensus.cc:729] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: LEARNER
I20240501 19:56:44.249635 18527 consensus_queue.cc:260] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:44.249903 18527 ts_tablet_manager.cc:1398] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:44.250998 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 10ee60ead7b94c93ab749ec16a04b0dc-3e16610fde0b47a199efc533627ae0ab received from {username='slave'} at 127.5.40.1:49437
I20240501 19:56:44.251070 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 10ee60ead7b94c93ab749ec16a04b0dc-3e16610fde0b47a199efc533627ae0ab on tablet 3e16610fde0b47a199efc533627ae0ab with peer 10ee60ead7b94c93ab749ec16a04b0dc
I20240501 19:56:44.272231 18540 ts_tablet_manager.cc:902] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: Initiating tablet copy from peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:44.272722 18540 tablet_copy_client.cc:308] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.7:36243
I20240501 19:56:44.273216 13952 tablet_copy_service.cc:140] P 6921cf2e41f64d66842885512da8de6c: Received BeginTabletCopySession request for tablet b85f5acd65914c2d8f1cc48d668c40eb from peer 8ad18cd54c6340fd92b874157875fa49 ({username='slave'} at 127.5.40.4:38499)
I20240501 19:56:44.273325 13952 tablet_copy_service.cc:161] P 6921cf2e41f64d66842885512da8de6c: Beginning new tablet copy session on tablet b85f5acd65914c2d8f1cc48d668c40eb from peer 8ad18cd54c6340fd92b874157875fa49 at {username='slave'} at 127.5.40.4:38499: session id = 8ad18cd54c6340fd92b874157875fa49-b85f5acd65914c2d8f1cc48d668c40eb
I20240501 19:56:44.274076 13952 tablet_copy_source_session.cc:205] T b85f5acd65914c2d8f1cc48d668c40eb P 6921cf2e41f64d66842885512da8de6c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.274938 18540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85f5acd65914c2d8f1cc48d668c40eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.277060 18540 tablet_copy_client.cc:783] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.277218 18540 tablet_copy_client.cc:647] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:44.277365 11097 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 19:56:44.279263 18540 tablet_copy_client.cc:523] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:44.280534 18540 tablet_bootstrap.cc:492] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:44.283042 18467 tablet.cc:1722] T 05c47f76981c418aad6e12afc32501ed P 6921cf2e41f64d66842885512da8de6c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:44.283516 18035 tablet.cc:1722] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:44.284076 18468 tablet.cc:1722] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 19:56:44.286868 18540 tablet_bootstrap.cc:492] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.287104 18540 tablet_bootstrap.cc:492] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: Bootstrap complete.
I20240501 19:56:44.287290 18540 ts_tablet_manager.cc:1367] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 19:56:44.287782 18540 raft_consensus.cc:348] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:44.287892 18540 raft_consensus.cc:729] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: LEARNER
I20240501 19:56:44.287994 18540 consensus_queue.cc:260] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:44.288336 18540 ts_tablet_manager.cc:1398] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.288132 11106 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: rebalancer_dynamic_table_new_name [id=3b03f2fb37964559b6161ed99fb86d81] alter complete (version 3)
I20240501 19:56:44.289561 13952 tablet_copy_service.cc:342] P 6921cf2e41f64d66842885512da8de6c: Request end of tablet copy session 8ad18cd54c6340fd92b874157875fa49-b85f5acd65914c2d8f1cc48d668c40eb received from {username='slave'} at 127.5.40.4:38499
I20240501 19:56:44.289646 13952 tablet_copy_service.cc:433] P 6921cf2e41f64d66842885512da8de6c: ending tablet copy session 8ad18cd54c6340fd92b874157875fa49-b85f5acd65914c2d8f1cc48d668c40eb on tablet b85f5acd65914c2d8f1cc48d668c40eb with peer 8ad18cd54c6340fd92b874157875fa49
I20240501 19:56:44.296461 18546 ts_tablet_manager.cc:902] T e6c1d3814e364e718449c3e4379b31ed P 6921cf2e41f64d66842885512da8de6c: Initiating tablet copy from peer ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
F20240501 19:56:44.296669 18546 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-00b7dd445e2846d5a0e1277e160d46bc.queue_time_us, tablet-download-pool-00b7dd445e2846d5a0e1277e160d46bc.queue_time_us) (tablet-download-pool-00b7dd445e2846d5a0e1277e160d46bc.run_cpu_time_us, tablet-download-pool-00b7dd445e2846d5a0e1277e160d46bc.run_cpu_time_us) (tablet-download-pool-00b7dd445e2846d5a0e1277e160d46bc.run_wall_time_us, tablet-download-pool-00b7dd445e2846d5a0e1277e160d46bc.run_wall_time_us) (tablet-download-pool-01a10926bce14cf1bd12f21287e08c89.queue_time_us, tablet-download-pool-01a10926bce14cf1bd12f21287e08c89.queue_time_us) (tablet-download-pool-01a10926bce14cf1bd12f21287e08c89.run_cpu_time_us, tablet-download-pool-01a10926bce14cf1bd12f21287e08c89.run_cpu_time_us) (tablet-download-pool-01a10926bce14cf1bd12f21287e08c89.run_wall_time_us, tablet-download-pool-01a10926bce14cf1bd12f21287e08c89.run_wall_time_us) (tablet-download-pool-1772073172764f8db95c597f8da14f12.queue_time_us, tablet-download-pool-1772073172764f8db95c597f8da14f12.queue_time_us) (tablet-download-pool-1772073172764f8db95c597f8da14f12.run_cpu_time_us, tablet-download-pool-1772073172764f8db95c597f8da14f12.run_cpu_time_us) (tablet-download-pool-1772073172764f8db95c597f8da14f12.run_wall_time_us, tablet-download-pool-1772073172764f8db95c597f8da14f12.run_wall_time_us) (tablet-download-pool-1c07d9fc05254a6399afcc54b3efb8ac.queue_time_us, tablet-download-pool-1c07d9fc05254a6399afcc54b3efb8ac.queue_time_us) (tablet-download-pool-1c07d9fc05254a6399afcc54b3efb8ac.run_cpu_time_us, tablet-download-pool-1c07d9fc05254a6399afcc54b3efb8ac.run_cpu_time_us) (tablet-download-pool-1c07d9fc05254a6399afcc54b3efb8ac.run_wall_time_us, tablet-download-pool-1c07d9fc05254a6399afcc54b3efb8ac.run_wall_time_us) (tablet-download-pool-1fc692e320d94825964040105a3175cd.queue_time_us, tablet-download-pool-1fc692e320d94825964040105a3175cd.queue_time_us) (tablet-download-pool-1fc692e320d94825964040105a3175cd.run_cpu_time_us, tablet-download-pool-1fc692e320d94825964040105a3175cd.run_cpu_time_us) (tablet-download-pool-1fc692e320d94825964040105a3175cd.run_wall_time_us, tablet-download-pool-1fc692e320d94825964040105a3175cd.run_wall_time_us) (tablet-download-pool-6070b62336054ef4b0444f27844b9ee1.queue_time_us, tablet-download-pool-6070b62336054ef4b0444f27844b9ee1.queue_time_us) (tablet-download-pool-6070b62336054ef4b0444f27844b9ee1.run_cpu_time_us, tablet-download-pool-6070b62336054ef4b0444f27844b9ee1.run_cpu_time_us) (tablet-download-pool-6070b62336054ef4b0444f27844b9ee1.run_wall_time_us, tablet-download-pool-6070b62336054ef4b0444f27844b9ee1.run_wall_time_us) (tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.queue_time_us, tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.queue_time_us) (tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.run_cpu_time_us, tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.run_cpu_time_us) (tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.run_wall_time_us, tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.run_wall_time_us) (tablet-download-pool-85ba57eb6aae4f12b4a4641d633104a0.queue_time_us, tablet-download-pool-85ba57eb6aae4f12b4a4641d633104a0.queue_time_us) (tablet-download-pool-85ba57eb6aae4f12b4a4641d633104a0.run_cpu_time_us, tablet-download-pool-85ba57eb6aae4f12b4a4641d633104a0.run_cpu_time_us) (tablet-download-pool-85ba57eb6aae4f12b4a4641d633104a0.run_wall_time_us, tablet-download-pool-85ba57eb6aae4f12b4a4641d633104a0.run_wall_time_us) (tablet-download-pool-8f3176bda2ad4b9986cda48e4a03ce49.queue_time_us, tablet-download-pool-8f3176bda2ad4b9986cda48e4a03ce49.queue_time_us) (tablet-download-pool-8f3176bda2ad4b9986cda48e4a03ce49.run_cpu_time_us, tablet-download-pool-8f3176bda2ad4b9986cda48e4a03ce49.run_cpu_time_us) (tablet-download-pool-8f3176bda2ad4b9986cda48e4a03ce49.run_wall_time_us, tablet-download-pool-8f3176bda2ad4b9986cda48e4a03ce49.run_wall_time_us) (tablet-download-pool-b7073666bc3f4c0a8c1e003fb1ef4fef.queue_time_us, tablet-download-pool-b7073666bc3f4c0a8c1e003fb1ef4fef.queue_time_us) (tablet-download-pool-b7073666bc3f4c0a8c1e003fb1ef4fef.run_cpu_time_us, tablet-download-pool-b7073666bc3f4c0a8c1e003fb1ef4fef.run_cpu_time_us) (tablet-download-pool-b7073666bc3f4c0a8c1e003fb1ef4fef.run_wall_time_us, tablet-download-pool-b7073666bc3f4c0a8c1e003fb1ef4fef.run_wall_time_us) (tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.queue_time_us, tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.queue_time_us) (tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.run_cpu_time_us, tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.run_cpu_time_us) (tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.run_wall_time_us, tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.run_wall_time_us) (tablet-download-pool-d505c415daa54005b8ed22bdc11ce135.queue_time_us, tablet-download-pool-d505c415daa54005b8ed22bdc11ce135.queue_time_us) (tablet-download-pool-d505c415daa54005b8ed22bdc11ce135.run_cpu_time_us, tablet-download-pool-d505c415daa54005b8ed22bdc11ce135.run_cpu_time_us) (tablet-download-pool-d505c415daa54005b8ed22bdc11ce135.run_wall_time_us, tablet-download-pool-d505c415daa54005b8ed22bdc11ce135.run_wall_time_us) (tablet-download-pool-e6c1d3814e364e718449c3e4379b31ed.queue_time_us, tablet-download-pool-e6c1d3814e364e718449c3e4379b31ed.queue_time_us) (tablet-download-pool-ea873ec8ced340b4ac082417c9b09aae.queue_time_us, tablet-download-pool-ea873ec8ced340b4ac082417c9b09aae.queue_time_us) (tablet-download-pool-ea873ec8ced340b4ac082417c9b09aae.run_cpu_time_us, tablet-download-pool-ea873ec8ced340b4ac082417c9b09aae.run_cpu_time_us) (tablet-download-pool-ea873ec8ced340b4ac082417c9b09aae.run_wall_time_us, tablet-download-pool-ea873ec8ced340b4ac082417c9b09aae.run_wall_time_us) (tablet-download-pool-eff09e5884f44e18a66583e8e0e73679.queue_time_us, tablet-download-pool-eff09e5884f44e18a66583e8e0e73679.queue_time_us) (tablet-download-pool-eff09e5884f44e18a66583e8e0e73679.run_cpu_time_us, tablet-download-pool-eff09e5884f44e18a66583e8e0e73679.run_cpu_time_us) (tablet-download-pool-eff09e5884f44e18a66583e8e0e73679.run_wall_time_us, tablet-download-pool-eff09e5884f44e18a66583e8e0e73679.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f0560263dcd google::LogMessage::Fail() at ??:0
@ 0x7f0560267b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f05602637cc google::LogMessage::Flush() at ??:0
@ 0x7f0560264f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f05616cd044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f05616b1f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f05616afefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f056426277a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f056426ba45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f05642ec76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f05643004f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f05642eb226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f05642fac0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x556bf1aa275c std::function<>::operator()() at ??:0
@ 0x7f05616b4e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f05616b56ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f05616b703b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x556bf1aa275c std::function<>::operator()() at ??:0
@ 0x7f05616a4807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0561ee26db start_thread at ??:0
@ 0x7f055f41a71f clone at ??:0
I20240501 19:56:44.371562 18522 ts_tablet_manager.cc:902] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:44.372013 18522 tablet_copy_client.cc:308] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.3:45965
I20240501 19:56:44.372540 11533 tablet_copy_service.cc:140] P 08711534813d47ba85aaec0771ed5d0f: Received BeginTabletCopySession request for tablet 33be1f7276c74d49b42c8761e3e4cd4e from peer ef60ab93b49c4346af4f62d3b98558b8 ({username='slave'} at 127.5.40.2:37613)
I20240501 19:56:44.372643 11533 tablet_copy_service.cc:161] P 08711534813d47ba85aaec0771ed5d0f: Beginning new tablet copy session on tablet 33be1f7276c74d49b42c8761e3e4cd4e from peer ef60ab93b49c4346af4f62d3b98558b8 at {username='slave'} at 127.5.40.2:37613: session id = ef60ab93b49c4346af4f62d3b98558b8-33be1f7276c74d49b42c8761e3e4cd4e
I20240501 19:56:44.373642 11533 tablet_copy_source_session.cc:205] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.375012 18522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33be1f7276c74d49b42c8761e3e4cd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.377465 18522 tablet_copy_client.cc:783] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.377593 18522 tablet_copy_client.cc:647] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:44.381387 18522 tablet_copy_client.cc:523] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:44.382704 18522 tablet_bootstrap.cc:492] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:44.387207 18522 tablet_bootstrap.cc:492] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.387471 18522 tablet_bootstrap.cc:492] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap complete.
I20240501 19:56:44.387620 18522 ts_tablet_manager.cc:1367] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240501 19:56:44.388047 18522 raft_consensus.cc:348] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:44.388159 18522 raft_consensus.cc:729] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: LEARNER
I20240501 19:56:44.388252 18522 consensus_queue.cc:260] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:44.389266 18522 ts_tablet_manager.cc:1398] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:44.390444 11533 tablet_copy_service.cc:342] P 08711534813d47ba85aaec0771ed5d0f: Request end of tablet copy session ef60ab93b49c4346af4f62d3b98558b8-33be1f7276c74d49b42c8761e3e4cd4e received from {username='slave'} at 127.5.40.2:37613
I20240501 19:56:44.390517 11533 tablet_copy_service.cc:433] P 08711534813d47ba85aaec0771ed5d0f: ending tablet copy session ef60ab93b49c4346af4f62d3b98558b8-33be1f7276c74d49b42c8761e3e4cd4e on tablet 33be1f7276c74d49b42c8761e3e4cd4e with peer ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:44.393605 11102 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:44.393730 11102 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46780:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 19:56:44.396461 11102 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 351e1e4f69a849a0b27128be4e72e5bb: Sending DeleteTablet for 3 replicas of tablet 05c47f76981c418aad6e12afc32501ed
I20240501 19:56:44.397188 14042 tablet_service.cc:1514] Processing DeleteTablet for tablet 05c47f76981c418aad6e12afc32501ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:44 UTC) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:44.397439 11491 tablet_service.cc:1514] Processing DeleteTablet for tablet 05c47f76981c418aad6e12afc32501ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:44 UTC) from {username='slave'} at 127.0.0.1:35094
I20240501 19:56:44.397542 18608 tablet_replica.cc:330] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:44.397679 18608 raft_consensus.cc:2229] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:44.397783 18608 raft_consensus.cc:2258] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:44.398226 18609 tablet_replica.cc:330] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: stopping tablet replica
I20240501 19:56:44.398365 18609 raft_consensus.cc:2229] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:44.398402 18608 ts_tablet_manager.cc:1875] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:44.398538 18609 raft_consensus.cc:2258] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:44.399984 18609 ts_tablet_manager.cc:1875] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:44.402462 18608 ts_tablet_manager.cc:1888] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:44.402531 18608 log.cc:1196] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/05c47f76981c418aad6e12afc32501ed
I20240501 19:56:44.402774 18608 ts_tablet_manager.cc:1909] T 05c47f76981c418aad6e12afc32501ed P 533908e517b24f77b2af8b5863e4ae5a: Deleting consensus metadata
I20240501 19:56:44.403429 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet 05c47f76981c418aad6e12afc32501ed (table rebalancer_dynamic_table_new_name [id=3b03f2fb37964559b6161ed99fb86d81]) successfully deleted
I20240501 19:56:44.405285 18609 ts_tablet_manager.cc:1888] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 19:56:44.405371 18609 log.cc:1196] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-2/wal/wals/05c47f76981c418aad6e12afc32501ed
I20240501 19:56:44.405694 18609 ts_tablet_manager.cc:1909] T 05c47f76981c418aad6e12afc32501ed P 08711534813d47ba85aaec0771ed5d0f: Deleting consensus metadata
I20240501 19:56:44.406412 11089 catalog_manager.cc:4830] TS 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965): tablet 05c47f76981c418aad6e12afc32501ed (table rebalancer_dynamic_table_new_name [id=3b03f2fb37964559b6161ed99fb86d81]) successfully deleted
I20240501 19:56:44.419312 18540 ts_tablet_manager.cc:902] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: Initiating tablet copy from peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:44.419757 18540 tablet_copy_client.cc:308] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.8:34789
I20240501 19:56:44.420337 14083 tablet_copy_service.cc:140] P 533908e517b24f77b2af8b5863e4ae5a: Received BeginTabletCopySession request for tablet 0bc4ce342be94b309a35450d3c59a6ce from peer 8ad18cd54c6340fd92b874157875fa49 ({username='slave'} at 127.5.40.4:33773)
I20240501 19:56:44.420431 14083 tablet_copy_service.cc:161] P 533908e517b24f77b2af8b5863e4ae5a: Beginning new tablet copy session on tablet 0bc4ce342be94b309a35450d3c59a6ce from peer 8ad18cd54c6340fd92b874157875fa49 at {username='slave'} at 127.5.40.4:33773: session id = 8ad18cd54c6340fd92b874157875fa49-0bc4ce342be94b309a35450d3c59a6ce
I20240501 19:56:44.421428 14083 tablet_copy_source_session.cc:205] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.422359 18540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc4ce342be94b309a35450d3c59a6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.424604 18540 tablet_copy_client.cc:783] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.424717 18540 tablet_copy_client.cc:647] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:44.425904 18540 tablet_copy_client.cc:523] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:44.427124 18540 tablet_bootstrap.cc:492] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: Bootstrap starting.
I20240501 19:56:44.430696 18540 tablet_bootstrap.cc:492] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.430938 18540 tablet_bootstrap.cc:492] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: Bootstrap complete.
I20240501 19:56:44.431098 18540 ts_tablet_manager.cc:1367] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:44.431584 18540 raft_consensus.cc:348] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:44.431715 18540 raft_consensus.cc:729] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Initialized, Role: LEARNER
I20240501 19:56:44.431823 18540 consensus_queue.cc:260] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:44.432116 18540 ts_tablet_manager.cc:1398] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:44.432709 14083 tablet_copy_service.cc:342] P 533908e517b24f77b2af8b5863e4ae5a: Request end of tablet copy session 8ad18cd54c6340fd92b874157875fa49-0bc4ce342be94b309a35450d3c59a6ce received from {username='slave'} at 127.5.40.4:33773
I20240501 19:56:44.432816 14083 tablet_copy_service.cc:433] P 533908e517b24f77b2af8b5863e4ae5a: ending tablet copy session 8ad18cd54c6340fd92b874157875fa49-0bc4ce342be94b309a35450d3c59a6ce on tablet 0bc4ce342be94b309a35450d3c59a6ce with peer 8ad18cd54c6340fd92b874157875fa49
I20240501 19:56:44.466747 18527 ts_tablet_manager.cc:902] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: Initiating tablet copy from peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:44.467300 18527 tablet_copy_client.cc:308] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.5:42501
I20240501 19:56:44.467904 13690 tablet_copy_service.cc:140] P 8125d8ad6d0d44caae9c5c90c90df96b: Received BeginTabletCopySession request for tablet ee5178fb00d741d49d42168a71e5d23a from peer 10ee60ead7b94c93ab749ec16a04b0dc ({username='slave'} at 127.5.40.1:50477)
I20240501 19:56:44.468019 13690 tablet_copy_service.cc:161] P 8125d8ad6d0d44caae9c5c90c90df96b: Beginning new tablet copy session on tablet ee5178fb00d741d49d42168a71e5d23a from peer 10ee60ead7b94c93ab749ec16a04b0dc at {username='slave'} at 127.5.40.1:50477: session id = 10ee60ead7b94c93ab749ec16a04b0dc-ee5178fb00d741d49d42168a71e5d23a
I20240501 19:56:44.468933 13690 tablet_copy_source_session.cc:205] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.469305 14794 consensus_queue.cc:1035] T 2d4cf91c71044cf6a37528e8c23954c0 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.470000 18527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5178fb00d741d49d42168a71e5d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.472555 18527 tablet_copy_client.cc:783] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.472822 18527 tablet_copy_client.cc:647] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:44.474771 18527 tablet_copy_client.cc:523] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:44.476505 18527 tablet_bootstrap.cc:492] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap starting.
I20240501 19:56:44.479372 18527 tablet_bootstrap.cc:492] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.479610 18527 tablet_bootstrap.cc:492] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: Bootstrap complete.
I20240501 19:56:44.479753 18527 ts_tablet_manager.cc:1367] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.001s
I20240501 19:56:44.480119 18527 raft_consensus.cc:348] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:44.480219 18527 raft_consensus.cc:729] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Initialized, Role: LEARNER
I20240501 19:56:44.480299 18527 consensus_queue.cc:260] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: NON_VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: true } }
I20240501 19:56:44.480571 18527 ts_tablet_manager.cc:1398] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.481392 13690 tablet_copy_service.cc:342] P 8125d8ad6d0d44caae9c5c90c90df96b: Request end of tablet copy session 10ee60ead7b94c93ab749ec16a04b0dc-ee5178fb00d741d49d42168a71e5d23a received from {username='slave'} at 127.5.40.1:50477
I20240501 19:56:44.481475 13690 tablet_copy_service.cc:433] P 8125d8ad6d0d44caae9c5c90c90df96b: ending tablet copy session 10ee60ead7b94c93ab749ec16a04b0dc-ee5178fb00d741d49d42168a71e5d23a on tablet ee5178fb00d741d49d42168a71e5d23a with peer 10ee60ead7b94c93ab749ec16a04b0dc
I20240501 19:56:44.486697 18528 ts_tablet_manager.cc:902] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: Initiating tablet copy from peer 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:44.487223 18528 tablet_copy_client.cc:308] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.8:34789
I20240501 19:56:44.487600 18646 ts_tablet_manager.cc:902] T 1fc7da07b6c5414bb305264962d0f218 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:44.487701 14083 tablet_copy_service.cc:140] P 533908e517b24f77b2af8b5863e4ae5a: Received BeginTabletCopySession request for tablet 66bdddd3550b4ae8a97535e4747a5234 from peer 08711534813d47ba85aaec0771ed5d0f ({username='slave'} at 127.5.40.3:49777)
I20240501 19:56:44.487816 14083 tablet_copy_service.cc:161] P 533908e517b24f77b2af8b5863e4ae5a: Beginning new tablet copy session on tablet 66bdddd3550b4ae8a97535e4747a5234 from peer 08711534813d47ba85aaec0771ed5d0f at {username='slave'} at 127.5.40.3:49777: session id = 08711534813d47ba85aaec0771ed5d0f-66bdddd3550b4ae8a97535e4747a5234
F20240501 19:56:44.487773 18646 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-01ed8143e5454a4aaef3e142069f35a2.queue_time_us, tablet-download-pool-01ed8143e5454a4aaef3e142069f35a2.queue_time_us) (tablet-download-pool-01ed8143e5454a4aaef3e142069f35a2.run_cpu_time_us, tablet-download-pool-01ed8143e5454a4aaef3e142069f35a2.run_cpu_time_us) (tablet-download-pool-01ed8143e5454a4aaef3e142069f35a2.run_wall_time_us, tablet-download-pool-01ed8143e5454a4aaef3e142069f35a2.run_wall_time_us) (tablet-download-pool-04932055883c412f944e839982815ab6.queue_time_us, tablet-download-pool-04932055883c412f944e839982815ab6.queue_time_us) (tablet-download-pool-04932055883c412f944e839982815ab6.run_cpu_time_us, tablet-download-pool-04932055883c412f944e839982815ab6.run_cpu_time_us) (tablet-download-pool-04932055883c412f944e839982815ab6.run_wall_time_us, tablet-download-pool-04932055883c412f944e839982815ab6.run_wall_time_us) (tablet-download-pool-1fc7da07b6c5414bb305264962d0f218.queue_time_us, tablet-download-pool-1fc7da07b6c5414bb305264962d0f218.queue_time_us) (tablet-download-pool-2c86f45702e94c0e898e4e81148f2098.queue_time_us, tablet-download-pool-2c86f45702e94c0e898e4e81148f2098.queue_time_us) (tablet-download-pool-2c86f45702e94c0e898e4e81148f2098.run_cpu_time_us, tablet-download-pool-2c86f45702e94c0e898e4e81148f2098.run_cpu_time_us) (tablet-download-pool-2c86f45702e94c0e898e4e81148f2098.run_wall_time_us, tablet-download-pool-2c86f45702e94c0e898e4e81148f2098.run_wall_time_us) (tablet-download-pool-3d99f4d3127348dba949ab5fd4666956.queue_time_us, tablet-download-pool-3d99f4d3127348dba949ab5fd4666956.queue_time_us) (tablet-download-pool-3d99f4d3127348dba949ab5fd4666956.run_cpu_time_us, tablet-download-pool-3d99f4d3127348dba949ab5fd4666956.run_cpu_time_us) (tablet-download-pool-3d99f4d3127348dba949ab5fd4666956.run_wall_time_us, tablet-download-pool-3d99f4d3127348dba949ab5fd4666956.run_wall_time_us) (tablet-download-pool-4dc0294d0dc5427e80411be5db9c95c1.queue_time_us, tablet-download-pool-4dc0294d0dc5427e80411be5db9c95c1.queue_time_us) (tablet-download-pool-4dc0294d0dc5427e80411be5db9c95c1.run_cpu_time_us, tablet-download-pool-4dc0294d0dc5427e80411be5db9c95c1.run_cpu_time_us) (tablet-download-pool-4dc0294d0dc5427e80411be5db9c95c1.run_wall_time_us, tablet-download-pool-4dc0294d0dc5427e80411be5db9c95c1.run_wall_time_us) (tablet-download-pool-582d72e630c84088a6e4a7309ae7e952.queue_time_us, tablet-download-pool-582d72e630c84088a6e4a7309ae7e952.queue_time_us) (tablet-download-pool-582d72e630c84088a6e4a7309ae7e952.run_cpu_time_us, tablet-download-pool-582d72e630c84088a6e4a7309ae7e952.run_cpu_time_us) (tablet-download-pool-582d72e630c84088a6e4a7309ae7e952.run_wall_time_us, tablet-download-pool-582d72e630c84088a6e4a7309ae7e952.run_wall_time_us) (tablet-download-pool-61cadb3ab334435eaebe99fc3c9e496b.queue_time_us, tablet-download-pool-61cadb3ab334435eaebe99fc3c9e496b.queue_time_us) (tablet-download-pool-61cadb3ab334435eaebe99fc3c9e496b.run_cpu_time_us, tablet-download-pool-61cadb3ab334435eaebe99fc3c9e496b.run_cpu_time_us) (tablet-download-pool-61cadb3ab334435eaebe99fc3c9e496b.run_wall_time_us, tablet-download-pool-61cadb3ab334435eaebe99fc3c9e496b.run_wall_time_us) (tablet-download-pool-6326c660e6cb4eb289634a68fc50c4af.queue_time_us, tablet-download-pool-6326c660e6cb4eb289634a68fc50c4af.queue_time_us) (tablet-download-pool-6326c660e6cb4eb289634a68fc50c4af.run_cpu_time_us, tablet-download-pool-6326c660e6cb4eb289634a68fc50c4af.run_cpu_time_us) (tablet-download-pool-6326c660e6cb4eb289634a68fc50c4af.run_wall_time_us, tablet-download-pool-6326c660e6cb4eb289634a68fc50c4af.run_wall_time_us) (tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.queue_time_us, tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.queue_time_us) (tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.run_cpu_time_us, tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.run_cpu_time_us) (tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.run_wall_time_us, tablet-download-pool-69facf86bba24f209c06afafc1ba8f7b.run_wall_time_us) (tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.queue_time_us, tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.queue_time_us) (tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.run_cpu_time_us, tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.run_cpu_time_us) (tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.run_wall_time_us, tablet-download-pool-c3b35dab7eec4b73b208f687dad46ba2.run_wall_time_us) (tablet-download-pool-cebd15701ee44b11a2f8804740fb4299.queue_time_us, tablet-download-pool-cebd15701ee44b11a2f8804740fb4299.queue_time_us) (tablet-download-pool-cebd15701ee44b11a2f8804740fb4299.run_cpu_time_us, tablet-download-pool-cebd15701ee44b11a2f8804740fb4299.run_cpu_time_us) (tablet-download-pool-cebd15701ee44b11a2f8804740fb4299.run_wall_time_us, tablet-download-pool-cebd15701ee44b11a2f8804740fb4299.run_wall_time_us) (tablet-download-pool-e6c1d3814e364e718449c3e4379b31ed.queue_time_us, tablet-download-pool-e6c1d3814e364e718449c3e4379b31ed.queue_time_us) (tablet-download-pool-e6c1d3814e364e718449c3e4379b31ed.run_cpu_time_us, tablet-download-pool-e6c1d3814e364e718449c3e4379b31ed.run_cpu_time_us) (tablet-download-pool-e6c1d3814e364e718449c3e4379b31ed.run_wall_time_us, tablet-download-pool-e6c1d3814e364e718449c3e4379b31ed.run_wall_time_us) (tablet-download-pool-e96e8e7c66584b1198bea464e5a1a4d8.queue_time_us, tablet-download-pool-e96e8e7c66584b1198bea464e5a1a4d8.queue_time_us) (tablet-download-pool-e96e8e7c66584b1198bea464e5a1a4d8.run_cpu_time_us, tablet-download-pool-e96e8e7c66584b1198bea464e5a1a4d8.run_cpu_time_us) (tablet-download-pool-e96e8e7c66584b1198bea464e5a1a4d8.run_wall_time_us, tablet-download-pool-e96e8e7c66584b1198bea464e5a1a4d8.run_wall_time_us) (tablet-download-pool-f0fb00c1acda4d98a90b12d877412045.queue_time_us, tablet-download-pool-f0fb00c1acda4d98a90b12d877412045.queue_time_us) (tablet-download-pool-f0fb00c1acda4d98a90b12d877412045.run_cpu_time_us, tablet-download-pool-f0fb00c1acda4d98a90b12d877412045.run_cpu_time_us) (tablet-download-pool-f0fb00c1acda4d98a90b12d877412045.run_wall_time_us, tablet-download-pool-f0fb00c1acda4d98a90b12d877412045.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 19:56:44.488562 14083 tablet_copy_source_session.cc:205] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.489774 18528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bdddd3550b4ae8a97535e4747a5234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.491091 18647 ts_tablet_manager.cc:902] T 6a5f03eecaec4b929723c8f02a185606 P 96fc18dbb5fd41afb9bebf805e8d33a7: Initiating tablet copy from peer 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
@ 0x7f70371aadcd google::LogMessage::Fail() at ??:0
I20240501 19:56:44.492240 18528 tablet_copy_client.cc:783] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.492383 18528 tablet_copy_client.cc:647] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 1 WAL segments...
@ 0x7f70371aeb93 google::LogMessage::SendToLog() at ??:0
I20240501 19:56:44.493897 18528 tablet_copy_client.cc:523] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f70371aa7cc google::LogMessage::Flush() at ??:0
I20240501 19:56:44.494980 18528 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
@ 0x7f70371abf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 19:56:44.498490 11106 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 19:56:44.498963 18528 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.499188 18528 tablet_bootstrap.cc:492] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap complete.
I20240501 19:56:44.499336 18528 ts_tablet_manager.cc:1367] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:44.500291 18528 raft_consensus.cc:348] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:44.500407 18528 raft_consensus.cc:729] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: LEARNER
I20240501 19:56:44.500516 18528 consensus_queue.cc:260] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:44.500778 18528 ts_tablet_manager.cc:1398] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:44.501642 14083 tablet_copy_service.cc:342] P 533908e517b24f77b2af8b5863e4ae5a: Request end of tablet copy session 08711534813d47ba85aaec0771ed5d0f-66bdddd3550b4ae8a97535e4747a5234 received from {username='slave'} at 127.5.40.3:49777
I20240501 19:56:44.501721 14083 tablet_copy_service.cc:433] P 533908e517b24f77b2af8b5863e4ae5a: ending tablet copy session 08711534813d47ba85aaec0771ed5d0f-66bdddd3550b4ae8a97535e4747a5234 on tablet 66bdddd3550b4ae8a97535e4747a5234 with peer 08711534813d47ba85aaec0771ed5d0f
@ 0x7f7038614044 kudu::TraceMetrics::InternName() at ??:0
I20240501 19:56:44.506728 13781 tablet_service.cc:1467] Processing CreateTablet for tablet ad5d7cd91f474de6a2ec6369e4776def (DEFAULT_TABLE table=rebalancer_dynamic_table [id=593ad1ce53514e608ffa0b271f5a19f2]), partition=
I20240501 19:56:44.506989 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5d7cd91f474de6a2ec6369e4776def. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f70385f8f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240501 19:56:44.509009 18651 tablet_bootstrap.cc:492] T ad5d7cd91f474de6a2ec6369e4776def P 96fc18dbb5fd41afb9bebf805e8d33a7: Bootstrap starting.
I20240501 19:56:44.509339 13650 tablet_service.cc:1467] Processing CreateTablet for tablet ad5d7cd91f474de6a2ec6369e4776def (DEFAULT_TABLE table=rebalancer_dynamic_table [id=593ad1ce53514e608ffa0b271f5a19f2]), partition=
I20240501 19:56:44.509575 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5d7cd91f474de6a2ec6369e4776def. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.510097 18651 tablet_bootstrap.cc:654] T ad5d7cd91f474de6a2ec6369e4776def P 96fc18dbb5fd41afb9bebf805e8d33a7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.511844 18652 tablet_bootstrap.cc:492] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:44.512702 18652 tablet_bootstrap.cc:654] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.513872 18652 tablet_bootstrap.cc:492] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:44.513950 18652 ts_tablet_manager.cc:1367] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:44.514389 18652 raft_consensus.cc:348] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:44.514520 18652 raft_consensus.cc:374] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.514554 18652 raft_consensus.cc:729] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.514675 18652 consensus_queue.cc:260] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:44.514977 18652 ts_tablet_manager.cc:1398] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.515719 18522 ts_tablet_manager.cc:902] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: Initiating tablet copy from peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:44.516165 18522 tablet_copy_client.cc:308] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.5:42501
I20240501 19:56:44.516760 13690 tablet_copy_service.cc:140] P 8125d8ad6d0d44caae9c5c90c90df96b: Received BeginTabletCopySession request for tablet fc5eb3e030774199836cff2a657e8d56 from peer ef60ab93b49c4346af4f62d3b98558b8 ({username='slave'} at 127.5.40.2:38435)
I20240501 19:56:44.516844 13690 tablet_copy_service.cc:161] P 8125d8ad6d0d44caae9c5c90c90df96b: Beginning new tablet copy session on tablet fc5eb3e030774199836cff2a657e8d56 from peer ef60ab93b49c4346af4f62d3b98558b8 at {username='slave'} at 127.5.40.2:38435: session id = ef60ab93b49c4346af4f62d3b98558b8-fc5eb3e030774199836cff2a657e8d56
I20240501 19:56:44.517733 13690 tablet_copy_source_session.cc:205] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f70385f6efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 19:56:44.519008 18522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc5eb3e030774199836cff2a657e8d56. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f703b1a977a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 19:56:44.521682 18522 tablet_copy_client.cc:783] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.521831 18522 tablet_copy_client.cc:647] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Starting download of 1 WAL segments...
@ 0x7f703b1b2a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 19:56:44.524147 18522 tablet_copy_client.cc:523] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:44.525592 18522 tablet_bootstrap.cc:492] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
@ 0x7f703b23376f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 19:56:44.530098 18522 tablet_bootstrap.cc:492] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
@ 0x7f703b2474f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 19:56:44.530334 18522 tablet_bootstrap.cc:492] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap complete.
I20240501 19:56:44.530506 18522 ts_tablet_manager.cc:1367] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 19:56:44.530973 18522 raft_consensus.cc:348] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
I20240501 19:56:44.531095 18522 raft_consensus.cc:729] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: LEARNER
I20240501 19:56:44.531193 18522 consensus_queue.cc:260] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: NON_VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: true } }
@ 0x7f703b232226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 19:56:44.534521 18522 ts_tablet_manager.cc:1398] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:44.536854 13690 tablet_copy_service.cc:342] P 8125d8ad6d0d44caae9c5c90c90df96b: Request end of tablet copy session ef60ab93b49c4346af4f62d3b98558b8-fc5eb3e030774199836cff2a657e8d56 received from {username='slave'} at 127.5.40.2:38435
I20240501 19:56:44.536937 13690 tablet_copy_service.cc:433] P 8125d8ad6d0d44caae9c5c90c90df96b: ending tablet copy session ef60ab93b49c4346af4f62d3b98558b8-fc5eb3e030774199836cff2a657e8d56 on tablet fc5eb3e030774199836cff2a657e8d56 with peer ef60ab93b49c4346af4f62d3b98558b8
@ 0x7f703b241c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55a54ec2575c std::function<>::operator()() at ??:0
I20240501 19:56:44.540810 18528 ts_tablet_manager.cc:902] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: Initiating tablet copy from peer 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:44.541294 18528 tablet_copy_client.cc:308] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.4:42041
@ 0x7f70385fbe06 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 19:56:44.541774 11664 tablet_copy_service.cc:140] P 8ad18cd54c6340fd92b874157875fa49: Received BeginTabletCopySession request for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 from peer 08711534813d47ba85aaec0771ed5d0f ({username='slave'} at 127.5.40.3:55181)
I20240501 19:56:44.541883 11664 tablet_copy_service.cc:161] P 8ad18cd54c6340fd92b874157875fa49: Beginning new tablet copy session on tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 from peer 08711534813d47ba85aaec0771ed5d0f at {username='slave'} at 127.5.40.3:55181: session id = 08711534813d47ba85aaec0771ed5d0f-9dad5cfa9b7a4a808eb9d0bb81f2a2c6
I20240501 19:56:44.542938 11664 tablet_copy_source_session.cc:205] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.543905 18528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.546286 18528 tablet_copy_client.cc:783] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.546429 18528 tablet_copy_client.cc:647] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 1 WAL segments...
@ 0x7f70385fc6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240501 19:56:44.550650 18528 tablet_copy_client.cc:523] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:44.552160 18528 tablet_bootstrap.cc:492] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
I20240501 19:56:44.555686 18528 tablet_bootstrap.cc:492] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.555917 18528 tablet_bootstrap.cc:492] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: Bootstrap complete.
I20240501 19:56:44.556047 18528 ts_tablet_manager.cc:1367] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 19:56:44.556541 18528 raft_consensus.cc:348] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:44.556650 18528 raft_consensus.cc:729] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: LEARNER
I20240501 19:56:44.556735 18528 consensus_queue.cc:260] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:44.557790 18528 ts_tablet_manager.cc:1398] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:44.558260 11664 tablet_copy_service.cc:342] P 8ad18cd54c6340fd92b874157875fa49: Request end of tablet copy session 08711534813d47ba85aaec0771ed5d0f-9dad5cfa9b7a4a808eb9d0bb81f2a2c6 received from {username='slave'} at 127.5.40.3:55181
I20240501 19:56:44.558344 11664 tablet_copy_service.cc:433] P 8ad18cd54c6340fd92b874157875fa49: ending tablet copy session 08711534813d47ba85aaec0771ed5d0f-9dad5cfa9b7a4a808eb9d0bb81f2a2c6 on tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 with peer 08711534813d47ba85aaec0771ed5d0f
I20240501 19:56:44.560811 18302 consensus_queue.cc:1035] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7f70385fe03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 19:56:44.564512 14794 consensus_queue.cc:1035] T 8efaba1013424acd949123503dc503b8 P 96fc18dbb5fd41afb9bebf805e8d33a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x55a54ec2575c std::function<>::operator()() at ??:0
@ 0x7f70385eb807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7038e296db start_thread at ??:0
@ 0x7f703636171f clone at ??:0
I20240501 19:56:44.574553 18528 ts_tablet_manager.cc:902] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: Initiating tablet copy from peer 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:44.575009 18528 tablet_copy_client.cc:308] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.40.5:42501
I20240501 19:56:44.575645 13690 tablet_copy_service.cc:140] P 8125d8ad6d0d44caae9c5c90c90df96b: Received BeginTabletCopySession request for tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d from peer 08711534813d47ba85aaec0771ed5d0f ({username='slave'} at 127.5.40.3:33027)
I20240501 19:56:44.575742 13690 tablet_copy_service.cc:161] P 8125d8ad6d0d44caae9c5c90c90df96b: Beginning new tablet copy session on tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d from peer 08711534813d47ba85aaec0771ed5d0f at {username='slave'} at 127.5.40.3:33027: session id = 08711534813d47ba85aaec0771ed5d0f-19d0d6aaa47e4f7eaa80ec5e0207d51d
I20240501 19:56:44.576654 13690 tablet_copy_source_session.cc:205] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:44.577447 18528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.579821 18528 tablet_copy_client.cc:783] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:44.579962 18528 tablet_copy_client.cc:647] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Starting download of 1 WAL segments...
W20240501 19:56:44.582818 13603 connection.cc:535] client connection to 127.5.40.7:36243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.582937 13603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.582953 11445 connection.cc:535] client connection to 127.5.40.7:36243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.583057 11445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.583696 11577 connection.cc:535] client connection to 127.5.40.7:36243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.583802 11577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.587817 11089 connection.cc:535] client connection to 127.5.40.7:36243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.587945 11089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.588482 11313 connection.cc:535] client connection to 127.5.40.7:36243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.588596 11313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.589217 11181 connection.cc:535] client connection to 127.5.40.7:36243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.589321 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.583402 13996 connection.cc:535] client connection to 127.5.40.7:36243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:56:44.589896 18528 tablet_copy_client.cc:523] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 19:56:44.589936 13996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.590015 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:44.590075 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.591228 18528 tablet_bootstrap.cc:492] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: Bootstrap starting.
W20240501 19:56:44.594753 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.596093 11089 catalog_manager.cc:4573] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05c47f76981c418aad6e12afc32501ed: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:44.596395 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.596546 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.596611 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.596649 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.597261 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.597327 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.597362 11577 consensus_peers.cc:485] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.597391 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.597896 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.597963 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.599521 18528 tablet_bootstrap.cc:492] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:44.599748 18528 tablet_bootstrap.cc:492] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: Bootstrap complete.
I20240501 19:56:44.599910 18528 ts_tablet_manager.cc:1367] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240501 19:56:44.600340 18528 raft_consensus.cc:348] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:44.600451 18528 raft_consensus.cc:729] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Initialized, Role: LEARNER
I20240501 19:56:44.600544 18528 consensus_queue.cc:260] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:44.600857 18528 ts_tablet_manager.cc:1398] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:44.602607 18208 raft_consensus.cc:482] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.602702 18208 raft_consensus.cc:504] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:44.603128 18208 leader_election.cc:288] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:44.603389 13690 tablet_copy_service.cc:342] P 8125d8ad6d0d44caae9c5c90c90df96b: Request end of tablet copy session 08711534813d47ba85aaec0771ed5d0f-19d0d6aaa47e4f7eaa80ec5e0207d51d received from {username='slave'} at 127.5.40.3:33027
I20240501 19:56:44.603475 13690 tablet_copy_service.cc:433] P 8125d8ad6d0d44caae9c5c90c90df96b: ending tablet copy session 08711534813d47ba85aaec0771ed5d0f-19d0d6aaa47e4f7eaa80ec5e0207d51d on tablet 19d0d6aaa47e4f7eaa80ec5e0207d51d with peer 08711534813d47ba85aaec0771ed5d0f
W20240501 19:56:44.609299 13603 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:44.621575 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.626056 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.698415 18340 consensus_queue.cc:1035] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.702492 16195 consensus_queue.cc:1035] T 2f09627997b5410a9d4f8149cf0f01b4 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:44.711725 13603 consensus_peers.cc:485] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.733157 11377 raft_consensus.cc:1203] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:44.739403 11102 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46780:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:44.746096 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 89c074e9edf94d8a8e422d972f0e6982 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4758dd74e48c4fb5a28bc2afd4f730ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.746245 11358 tablet_service.cc:1467] Processing CreateTablet for tablet b73565839e3342e785237ab4ee96c5c9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4758dd74e48c4fb5a28bc2afd4f730ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.746417 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c074e9edf94d8a8e422d972f0e6982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.746665 14042 tablet_service.cc:1467] Processing CreateTablet for tablet 89c074e9edf94d8a8e422d972f0e6982 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4758dd74e48c4fb5a28bc2afd4f730ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.746920 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c074e9edf94d8a8e422d972f0e6982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.749085 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 5657f0c0f383445f9c63d6afffbe5bd6 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4758dd74e48c4fb5a28bc2afd4f730ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.749114 18726 tablet_bootstrap.cc:492] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
I20240501 19:56:44.749358 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5657f0c0f383445f9c63d6afffbe5bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.751677 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 5657f0c0f383445f9c63d6afffbe5bd6 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4758dd74e48c4fb5a28bc2afd4f730ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.751380 18726 tablet_bootstrap.cc:654] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.751950 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5657f0c0f383445f9c63d6afffbe5bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.752486 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73565839e3342e785237ab4ee96c5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.753600 18726 tablet_bootstrap.cc:492] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:44.753679 18726 ts_tablet_manager.cc:1367] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:44.754127 18652 tablet_bootstrap.cc:492] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b: Bootstrap starting.
I20240501 19:56:44.754114 18726 raft_consensus.cc:348] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.754215 18726 raft_consensus.cc:374] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.754242 18726 raft_consensus.cc:729] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.754384 18726 consensus_queue.cc:260] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.754963 18726 ts_tablet_manager.cc:1398] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.755054 18652 tablet_bootstrap.cc:654] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b: Neither blocks nor log segments found. Creating new log.
W20240501 19:56:44.755740 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.755105 18726 tablet_bootstrap.cc:492] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8: Bootstrap starting.
W20240501 19:56:44.756254 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.756974 18726 tablet_bootstrap.cc:654] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.757004 18727 tablet_bootstrap.cc:492] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:44.758006 18727 tablet_bootstrap.cc:654] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.758910 18652 tablet_bootstrap.cc:492] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b: No bootstrap required, opened a new log
I20240501 19:56:44.759003 18652 ts_tablet_manager.cc:1367] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:44.759174 18727 tablet_bootstrap.cc:492] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:44.759245 18727 ts_tablet_manager.cc:1367] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:44.759367 18652 raft_consensus.cc:348] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.759474 18652 raft_consensus.cc:374] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.759503 18652 raft_consensus.cc:729] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.759617 18652 consensus_queue.cc:260] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.759678 18727 raft_consensus.cc:348] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.759773 18727 raft_consensus.cc:374] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.759799 18727 raft_consensus.cc:729] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.759907 18652 ts_tablet_manager.cc:1398] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.759915 18727 consensus_queue.cc:260] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.760151 18727 ts_tablet_manager.cc:1398] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.760308 18727 tablet_bootstrap.cc:492] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a: Bootstrap starting.
I20240501 19:56:44.761106 18727 tablet_bootstrap.cc:654] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.762209 18727 tablet_bootstrap.cc:492] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a: No bootstrap required, opened a new log
I20240501 19:56:44.762290 18727 ts_tablet_manager.cc:1367] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:44.762676 18727 raft_consensus.cc:348] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.762778 18727 raft_consensus.cc:374] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.762812 18727 raft_consensus.cc:729] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.762943 18727 consensus_queue.cc:260] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.763209 18727 ts_tablet_manager.cc:1398] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 19:56:44.766708 11184 connection.cc:535] client connection to 127.5.40.6:46331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.768546 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.769435 18726 tablet_bootstrap.cc:492] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8: No bootstrap required, opened a new log
I20240501 19:56:44.769551 18726 ts_tablet_manager.cc:1367] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 19:56:44.770120 18726 raft_consensus.cc:348] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:44.770279 18726 raft_consensus.cc:374] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.770324 18726 raft_consensus.cc:729] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.770454 18726 consensus_queue.cc:260] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:44.770727 18726 ts_tablet_manager.cc:1398] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240501 19:56:44.770893 13999 connection.cc:535] client connection to 127.5.40.6:46331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.770910 11316 connection.cc:535] client connection to 127.5.40.6:46331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.771020 13606 connection.cc:535] client connection to 127.5.40.6:46331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.772153 11448 connection.cc:535] client connection to 127.5.40.6:46331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.772585 11092 connection.cc:535] client connection to 127.5.40.6:46331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:44.773208 18693 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:44.771460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.40.6:46331 (local address 127.5.40.8:54921)
0501 19:56:44.771633 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.771647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.771728 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.772802 (+ 1074us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":100}
W20240501 19:56:44.773208 18705 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:44.771736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.40.6:46331 (local address 127.5.40.5:33219)
0501 19:56:44.771909 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.771920 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.771999 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.772669 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":101}
W20240501 19:56:44.773335 13606 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:56:44.773355 18307 raft_consensus.cc:482] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.773408 13606 leader_election.cc:302] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:44.773429 18307 raft_consensus.cc:504] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.773638 18208 raft_consensus.cc:2726] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:44.773684 18680 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:44.772616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.40.6:46331 (local address 127.5.40.3:59839)
0501 19:56:44.773168 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.773208 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":475}
I20240501 19:56:44.773747 18307 leader_election.cc:288] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
W20240501 19:56:44.773797 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection reset by peer (error 104)
W20240501 19:56:44.773850 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.773913 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection reset by peer (error 104)
W20240501 19:56:44.774286 13999 consensus_peers.cc:485] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.774353 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.774549 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c074e9edf94d8a8e422d972f0e6982" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:44.774669 14063 raft_consensus.cc:2454] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 0.
I20240501 19:56:44.774976 11313 leader_election.cc:302] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, ef60ab93b49c4346af4f62d3b98558b8; no voters:
W20240501 19:56:44.775110 11313 leader_election.cc:334] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:44.775110 18307 raft_consensus.cc:2781] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.775184 18307 raft_consensus.cc:482] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.775216 18307 raft_consensus.cc:3037] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:56:44.775636 18683 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:44.771805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.40.6:46331 (local address 127.5.40.2:58551)
0501 19:56:44.771918 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.771930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.772009 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.775227 (+ 3218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":38}
W20240501 19:56:44.775754 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.776201 18307 raft_consensus.cc:504] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:44.776496 18307 leader_election.cc:288] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
W20240501 19:56:44.776695 18691 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:44.767284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.40.6:46331 (local address 127.5.40.1:52861)
0501 19:56:44.767557 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.767570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.767663 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.776276 (+ 8613us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":60,"threads_started":1}
W20240501 19:56:44.776863 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.776926 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.776961 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.776952 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c074e9edf94d8a8e422d972f0e6982" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:44.777050 14063 raft_consensus.cc:3037] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.777930 14063 raft_consensus.cc:2454] T 89c074e9edf94d8a8e422d972f0e6982 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
W20240501 19:56:44.778005 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.778683 11313 leader_election.cc:334] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:44.778820 11313 leader_election.cc:302] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:44.779044 18307 raft_consensus.cc:2781] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.779119 18307 raft_consensus.cc:686] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:44.779228 18307 consensus_queue.cc:237] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
W20240501 19:56:44.779410 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.781163 11102 catalog_manager.cc:5484] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 1 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:44.796411 11377 raft_consensus.cc:1203] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 19:56:44.808923 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.829509 11644 raft_consensus.cc:1203] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:44.830631 11247 raft_consensus.cc:1203] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 19:56:44.871776 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:44.935361 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:44.939572 11511 raft_consensus.cc:1203] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 19:56:44.961501 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:44.970615 11377 raft_consensus.cc:1203] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 19:56:44.980193 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.996968 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.997081 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:44.999612 13999 consensus_peers.cc:485] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:45.000871 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.002707 16220 raft_consensus.cc:482] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.002811 16220 raft_consensus.cc:504] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:45.003182 16220 leader_election.cc:288] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:45.004144 11316 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:45.004221 11313 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.004279 11313 leader_election.cc:302] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.004417 16220 raft_consensus.cc:2726] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:45.008224 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:45.024494 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.029844 11511 raft_consensus.cc:1203] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 19:56:45.046506 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.053925 18721 raft_consensus.cc:482] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.054067 18721 raft_consensus.cc:504] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:45.054453 18721 leader_election.cc:288] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.054868 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5657f0c0f383445f9c63d6afffbe5bd6" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:45.055011 13669 raft_consensus.cc:2454] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 0.
I20240501 19:56:45.055480 13996 leader_election.cc:302] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
W20240501 19:56:45.055630 13996 leader_election.cc:334] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.055634 18721 raft_consensus.cc:2781] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.055730 18721 raft_consensus.cc:482] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.055773 18721 raft_consensus.cc:3037] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.056689 18721 raft_consensus.cc:504] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:45.057011 18721 leader_election.cc:288] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.057368 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5657f0c0f383445f9c63d6afffbe5bd6" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:45.057480 13669 raft_consensus.cc:3037] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:56:45.058079 13996 leader_election.cc:334] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.058388 13669 raft_consensus.cc:2454] T 5657f0c0f383445f9c63d6afffbe5bd6 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:45.058660 13996 leader_election.cc:302] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:45.058805 18721 raft_consensus.cc:2781] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.058883 18721 raft_consensus.cc:686] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:45.059039 18721 consensus_queue.cc:237] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } }
I20240501 19:56:45.061021 11098 catalog_manager.cc:5484] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.064865 11247 raft_consensus.cc:1203] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 19:56:45.132937 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.133878 11511 raft_consensus.cc:1203] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 19:56:45.291780 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.297683 18778 consensus_queue.cc:1035] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.314177 18721 raft_consensus.cc:482] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.314303 18721 raft_consensus.cc:504] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:45.314725 18721 leader_election.cc:288] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965)
I20240501 19:56:45.315372 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
W20240501 19:56:45.315778 13996 leader_election.cc:334] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.315949 13996 leader_election.cc:302] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 08711534813d47ba85aaec0771ed5d0f, 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:45.316109 18721 raft_consensus.cc:2726] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.317075 18126 raft_consensus.cc:1050] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 to VOTER
I20240501 19:56:45.317466 18126 consensus_queue.cc:237] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.318961 11377 raft_consensus.cc:1261] T 6caecf09f4114855bddc470eb96bd5af P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.319389 18724 consensus_queue.cc:1035] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 19:56:45.319590 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:45.319684 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.335965 18777 raft_consensus.cc:482] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.336118 18777 raft_consensus.cc:504] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:45.336545 18777 leader_election.cc:288] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:45.337002 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85f5acd65914c2d8f1cc48d668c40eb" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
W20240501 19:56:45.337572 13603 leader_election.cc:334] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.337639 13603 leader_election.cc:302] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:45.337775 18777 raft_consensus.cc:2726] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.338521 18724 raft_consensus.cc:1050] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc to VOTER
I20240501 19:56:45.338871 18724 consensus_queue.cc:237] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } }
I20240501 19:56:45.340229 13669 raft_consensus.cc:1261] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.340252 11247 raft_consensus.cc:1261] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.340340 14063 raft_consensus.cc:1261] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.340595 18787 consensus_queue.cc:1035] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:45.340806 18724 consensus_queue.cc:1035] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:45.340983 18126 consensus_queue.cc:1035] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:45.343070 18126 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.343607 11247 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.345305 13669 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.346441 11098 catalog_manager.cc:5484] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.347998 14063 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.350518 11644 consensus_queue.cc:237] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } }
I20240501 19:56:45.351943 13669 raft_consensus.cc:1261] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.352113 11247 raft_consensus.cc:1261] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.352938 18126 consensus_queue.cc:1035] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.353267 18126 consensus_queue.cc:1035] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.354480 18126 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.355998 13669 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.356451 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e16610fde0b47a199efc533627ae0ab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:45.356962 11247 raft_consensus.cc:2932] T 3e16610fde0b47a199efc533627ae0ab P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.357149 11098 catalog_manager.cc:5484] T 3e16610fde0b47a199efc533627ae0ab P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 3 to 4, VOTER 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.359880 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e16610fde0b47a199efc533627ae0ab with delete_type TABLET_DATA_TOMBSTONED (TS 533908e517b24f77b2af8b5863e4ae5a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:45.360663 18371 raft_consensus.cc:1050] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f: attempting to promote NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 to VOTER
I20240501 19:56:45.361122 18371 consensus_queue.cc:237] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.363011 11377 raft_consensus.cc:1261] T 33be1f7276c74d49b42c8761e3e4cd4e P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.363528 18793 consensus_queue.cc:1035] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 19:56:45.363986 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:45.364056 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.367102 18791 tablet_replica.cc:330] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a: stopping tablet replica
I20240501 19:56:45.367259 18791 raft_consensus.cc:2229] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.367404 18791 raft_consensus.cc:2258] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.367964 18791 ts_tablet_manager.cc:1875] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:45.370177 18791 ts_tablet_manager.cc:1888] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:45.370244 18791 log.cc:1196] T 3e16610fde0b47a199efc533627ae0ab P 533908e517b24f77b2af8b5863e4ae5a: Deleting WAL directory at /tmp/dist-test-tasktBk3Nr/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714593353431859-5280-0/raft_consensus-itest-cluster/ts-7/wal/wals/3e16610fde0b47a199efc533627ae0ab
I20240501 19:56:45.370747 11089 catalog_manager.cc:4830] TS 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789): tablet 3e16610fde0b47a199efc533627ae0ab (table rebalancer_extra_table_20 [id=5896ed302eed4b558d240eecb91f7d8a]) successfully deleted
I20240501 19:56:45.377197 18721 raft_consensus.cc:1050] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a: attempting to promote NON_VOTER 08711534813d47ba85aaec0771ed5d0f to VOTER
I20240501 19:56:45.377589 18721 consensus_queue.cc:237] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:45.379299 11511 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:45.379289 11644 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:45.379896 16195 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:56:45.380367 13999 consensus_peers.cc:485] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.380506 18797 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:45.382445 18721 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.382841 11511 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.384780 18793 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.384881 18793 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:45.384776 11643 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.385206 18793 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.385609 11098 catalog_manager.cc:5484] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 5 to 6, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 19:56:45.388962 11448 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.389043 14063 consensus_queue.cc:237] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
W20240501 19:56:45.390046 11445 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.390122 11445 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.390252 18793 raft_consensus.cc:2726] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:56:45.390331 11643 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:45.390539 11511 raft_consensus.cc:1261] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:45.390678 16195 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:45.390872 18340 consensus_queue.cc:1035] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:45.392118 16195 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.392885 11643 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.392866 11511 raft_consensus.cc:2932] T 66bdddd3550b4ae8a97535e4747a5234 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.393462 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66bdddd3550b4ae8a97535e4747a5234 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:45.395542 11102 catalog_manager.cc:5484] T 66bdddd3550b4ae8a97535e4747a5234 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 6 to 7, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.407555 18775 raft_consensus.cc:482] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.407689 18775 raft_consensus.cc:504] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:45.408152 18775 leader_election.cc:288] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.408574 11377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bb9ef27c6e4f248a0e5a4812986f38" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
W20240501 19:56:45.409001 11181 leader_election.cc:334] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.409070 11181 leader_election.cc:302] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc; no voters: 6921cf2e41f64d66842885512da8de6c, ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:45.409186 18775 raft_consensus.cc:2726] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.425730 18209 raft_consensus.cc:1050] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b: attempting to promote NON_VOTER 08711534813d47ba85aaec0771ed5d0f to VOTER
I20240501 19:56:45.426111 18209 consensus_queue.cc:237] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
W20240501 19:56:45.427879 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.428043 11511 raft_consensus.cc:1261] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240501 19:56:45.428175 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.428504 18801 consensus_queue.cc:1035] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 19:56:45.428660 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:45.435640 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.448676 18721 raft_consensus.cc:1050] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a: attempting to promote NON_VOTER 8ad18cd54c6340fd92b874157875fa49 to VOTER
I20240501 19:56:45.449074 18721 consensus_queue.cc:237] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:45.450446 11643 raft_consensus.cc:1261] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.450457 11377 raft_consensus.cc:1261] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.450918 16195 consensus_queue.cc:1035] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 19:56:45.451043 13999 consensus_peers.cc:485] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.451054 18797 consensus_queue.cc:1035] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:45.453455 18797 raft_consensus.cc:2932] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:45.455047 11643 raft_consensus.cc:2932] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:45.456537 11377 raft_consensus.cc:2932] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:45.457108 11100 catalog_manager.cc:5484] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 2 to 3, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.460223 14063 consensus_queue.cc:237] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:45.461357 11377 raft_consensus.cc:1261] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.461364 11643 raft_consensus.cc:1261] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.461745 18340 consensus_queue.cc:1035] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.462042 16195 consensus_queue.cc:1035] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.463532 18721 raft_consensus.cc:2932] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:45.464301 11377 raft_consensus.cc:2932] T 0bc4ce342be94b309a35450d3c59a6ce P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:45.464912 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bc4ce342be94b309a35450d3c59a6ce with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:45.465032 11643 raft_consensus.cc:2932] T 0bc4ce342be94b309a35450d3c59a6ce P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:45.466104 11102 catalog_manager.cc:5484] T 0bc4ce342be94b309a35450d3c59a6ce P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 3 to 4, VOTER 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) evicted. New cstate: current_term: 1 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.467898 18778 raft_consensus.cc:482] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.467988 18778 raft_consensus.cc:504] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:45.468315 18778 leader_election.cc:288] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:45.468876 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bb9ef27c6e4f248a0e5a4812986f38" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:45.469018 11247 raft_consensus.cc:2454] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:45.469568 11313 leader_election.cc:302] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
I20240501 19:56:45.469730 18778 raft_consensus.cc:2781] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:45.469795 18778 raft_consensus.cc:482] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.469847 18778 raft_consensus.cc:3037] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:45.470021 11313 leader_election.cc:334] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.470676 18778 raft_consensus.cc:504] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:45.470973 18778 leader_election.cc:288] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:45.471469 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7bb9ef27c6e4f248a0e5a4812986f38" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:45.471588 11247 raft_consensus.cc:3037] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:45.472456 11247 raft_consensus.cc:2454] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 2.
I20240501 19:56:45.473197 11313 leader_election.cc:302] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
W20240501 19:56:45.473354 11313 leader_election.cc:334] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.473361 18778 raft_consensus.cc:2781] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:45.473467 18778 raft_consensus.cc:686] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:45.473594 18778 consensus_queue.cc:237] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } }
I20240501 19:56:45.475628 11100 catalog_manager.cc:5484] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 1 to 2, leader changed from 6921cf2e41f64d66842885512da8de6c (127.5.40.7) to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 2 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.491504 18307 raft_consensus.cc:482] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.491621 18307 raft_consensus.cc:504] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.491936 18307 leader_election.cc:288] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965)
I20240501 19:56:45.492347 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8078e0cbffa943c98584224d05b0ece0" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
W20240501 19:56:45.492787 11316 leader_election.cc:334] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.492857 11316 leader_election.cc:302] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef60ab93b49c4346af4f62d3b98558b8; no voters: 10ee60ead7b94c93ab749ec16a04b0dc, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.492997 18307 raft_consensus.cc:2726] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.508466 18340 raft_consensus.cc:482] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.508636 18340 raft_consensus.cc:504] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:45.509120 18340 leader_election.cc:288] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4:42041)
I20240501 19:56:45.509546 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313fb8dab2644b339cdd22647a8550fc" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8ad18cd54c6340fd92b874157875fa49" is_pre_election: true
W20240501 19:56:45.510105 13996 leader_election.cc:334] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.510176 13996 leader_election.cc:302] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c, 8ad18cd54c6340fd92b874157875fa49
I20240501 19:56:45.510303 18340 raft_consensus.cc:2726] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.530635 18777 raft_consensus.cc:1050] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b: attempting to promote NON_VOTER ef60ab93b49c4346af4f62d3b98558b8 to VOTER
I20240501 19:56:45.531040 18777 consensus_queue.cc:237] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.532526 14063 raft_consensus.cc:1261] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.532991 18801 consensus_queue.cc:1035] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 19:56:45.533171 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.533185 11377 raft_consensus.cc:1261] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 LEARNER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.533550 18801 consensus_queue.cc:1035] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:45.536832 18209 raft_consensus.cc:2932] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:45.537263 11380 raft_consensus.cc:2932] T fc5eb3e030774199836cff2a657e8d56 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:45.539394 14062 raft_consensus.cc:2932] T fc5eb3e030774199836cff2a657e8d56 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } } }
I20240501 19:56:45.540587 11100 catalog_manager.cc:5484] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: config changed from index 2 to 3, ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 19:56:45.543712 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:45.587706 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.597462 18340 raft_consensus.cc:482] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.597570 18340 raft_consensus.cc:504] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:45.597988 18340 leader_election.cc:288] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
I20240501 19:56:45.598356 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9c3a48f92e40f19d75f4151bb4c3e3" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
W20240501 19:56:45.598922 13999 leader_election.cc:334] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.599022 13996 leader_election.cc:302] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 8125d8ad6d0d44caae9c5c90c90df96b, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.599176 18340 raft_consensus.cc:2726] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.624679 18126 raft_consensus.cc:1050] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49: attempting to promote NON_VOTER 08711534813d47ba85aaec0771ed5d0f to VOTER
I20240501 19:56:45.625110 18126 consensus_queue.cc:237] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:45.626425 11511 raft_consensus.cc:1261] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.626897 14062 raft_consensus.cc:1261] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240501 19:56:45.627040 11577 consensus_peers.cc:485] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.627354 18724 consensus_queue.cc:1035] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:45.627523 18787 consensus_queue.cc:1035] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:45.629639 18724 raft_consensus.cc:2932] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.630095 14062 raft_consensus.cc:2932] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.632656 11098 catalog_manager.cc:5484] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.632634 11511 raft_consensus.cc:2932] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.636230 11643 consensus_queue.cc:237] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:45.637809 11511 raft_consensus.cc:1261] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.638139 18126 consensus_queue.cc:1035] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.638729 14062 raft_consensus.cc:1261] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.639155 18126 consensus_queue.cc:1035] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.639729 18787 raft_consensus.cc:2932] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.640220 11511 raft_consensus.cc:2932] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:45.642068 11090 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:45.643173 11102 catalog_manager.cc:5484] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New cstate: current_term: 1 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.643503 14063 raft_consensus.cc:2932] T 9dad5cfa9b7a4a808eb9d0bb81f2a2c6 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
W20240501 19:56:45.646692 11089 catalog_manager.cc:4573] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.664901 18340 consensus_queue.cc:1035] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.670960 18778 raft_consensus.cc:482] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.671100 18778 raft_consensus.cc:504] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.671532 18778 leader_election.cc:288] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:45.671869 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aabbb9dc2440f28a3b6944dea38fe7" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
W20240501 19:56:45.672484 11316 leader_election.cc:334] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.672602 11316 leader_election.cc:302] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef60ab93b49c4346af4f62d3b98558b8; no voters: 10ee60ead7b94c93ab749ec16a04b0dc, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.672744 18778 raft_consensus.cc:2726] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.686983 18801 raft_consensus.cc:1050] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b: attempting to promote NON_VOTER 10ee60ead7b94c93ab749ec16a04b0dc to VOTER
I20240501 19:56:45.687407 18801 consensus_queue.cc:237] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } }
I20240501 19:56:45.688992 11247 raft_consensus.cc:1261] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 LEARNER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.689054 14063 raft_consensus.cc:1261] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:45.689393 18209 consensus_queue.cc:1035] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 19:56:45.689536 13603 consensus_peers.cc:485] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:45.689640 18777 consensus_queue.cc:1035] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:45.689708 18775 raft_consensus.cc:482] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.689800 18775 raft_consensus.cc:504] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:45.690162 18775 leader_election.cc:288] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:45.690595 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8fedf0883b4b5ebee7e028a94a503b" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
W20240501 19:56:45.691414 11181 leader_election.cc:334] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.691504 11181 leader_election.cc:302] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc; no voters: 6921cf2e41f64d66842885512da8de6c, ef60ab93b49c4346af4f62d3b98558b8
I20240501 19:56:45.691620 18775 raft_consensus.cc:2726] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.691957 18209 raft_consensus.cc:2932] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.692421 11247 raft_consensus.cc:2932] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.694084 14063 raft_consensus.cc:2932] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.695101 11100 catalog_manager.cc:5484] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: config changed from index 2 to 3, 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { replace: true } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.698102 13669 consensus_queue.cc:237] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } }
I20240501 19:56:45.699188 11247 raft_consensus.cc:1261] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.699481 14063 raft_consensus.cc:1261] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:45.699734 18208 consensus_queue.cc:1035] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.699859 18801 consensus_queue.cc:1035] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:45.701217 18209 raft_consensus.cc:2932] T ee5178fb00d741d49d42168a71e5d23a P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.701740 14063 raft_consensus.cc:2932] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.703308 11247 raft_consensus.cc:2932] T ee5178fb00d741d49d42168a71e5d23a P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.709214 11102 catalog_manager.cc:5484] T ee5178fb00d741d49d42168a71e5d23a P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 3 to 4, VOTER 6921cf2e41f64d66842885512da8de6c (127.5.40.7) evicted. New cstate: current_term: 1 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } }
I20240501 19:56:45.709566 11089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee5178fb00d741d49d42168a71e5d23a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:45.719113 18793 raft_consensus.cc:482] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.719218 18793 raft_consensus.cc:504] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:45.719662 18793 leader_election.cc:288] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:45.720307 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:45.720454 14063 raft_consensus.cc:2454] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 1.
W20240501 19:56:45.720521 11445 leader_election.cc:334] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.720700 11445 leader_election.cc:302] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:45.720899 18793 raft_consensus.cc:2781] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:45.720943 18793 raft_consensus.cc:482] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.720989 18793 raft_consensus.cc:3037] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:45.721896 18793 raft_consensus.cc:504] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:45.722196 18793 leader_election.cc:288] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 election: Requested vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:45.722702 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b4dda83bc408eaddec5ebc7b734d5" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:45.722822 14063 raft_consensus.cc:3037] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:45.723289 11445 leader_election.cc:334] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.724306 14063 raft_consensus.cc:2454] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08711534813d47ba85aaec0771ed5d0f in term 2.
I20240501 19:56:45.724578 11445 leader_election.cc:302] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:45.724769 18793 raft_consensus.cc:2781] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:45.724844 18793 raft_consensus.cc:686] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [term 2 LEADER]: Becoming Leader. State: Replica: 08711534813d47ba85aaec0771ed5d0f, State: Running, Role: LEADER
I20240501 19:56:45.724982 18793 consensus_queue.cc:237] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:45.726943 11100 catalog_manager.cc:5484] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f reported cstate change: term changed from 1 to 2, leader changed from 6921cf2e41f64d66842885512da8de6c (127.5.40.7) to 08711534813d47ba85aaec0771ed5d0f (127.5.40.3). New cstate: current_term: 2 leader_uuid: "08711534813d47ba85aaec0771ed5d0f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.746230 18778 raft_consensus.cc:482] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.746380 18778 raft_consensus.cc:504] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:45.746476 18724 raft_consensus.cc:482] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.746591 18724 raft_consensus.cc:504] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:45.746865 18778 leader_election.cc:288] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.746934 18724 leader_election.cc:288] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.747318 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8fedf0883b4b5ebee7e028a94a503b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:45.747529 11247 raft_consensus.cc:2454] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:45.747604 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313fb8dab2644b339cdd22647a8550fc" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:45.747751 14063 raft_consensus.cc:2454] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 1.
I20240501 19:56:45.748104 11313 leader_election.cc:302] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
W20240501 19:56:45.748176 11577 leader_election.cc:334] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.748277 18778 raft_consensus.cc:2781] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:45.748332 18778 raft_consensus.cc:482] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.748349 11577 leader_election.cc:302] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8ad18cd54c6340fd92b874157875fa49; no voters: 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:45.748384 18778 raft_consensus.cc:3037] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:45.748457 11313 leader_election.cc:334] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.748504 18724 raft_consensus.cc:2781] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:45.748553 18724 raft_consensus.cc:482] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.748589 18724 raft_consensus.cc:3037] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:45.749351 18778 raft_consensus.cc:504] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:45.749449 18724 raft_consensus.cc:504] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:45.749680 18778 leader_election.cc:288] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.749783 18724 leader_election.cc:288] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 election: Requested vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.750103 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313fb8dab2644b339cdd22647a8550fc" candidate_uuid: "8ad18cd54c6340fd92b874157875fa49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:45.750222 14063 raft_consensus.cc:3037] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:45.750782 11577 leader_election.cc:334] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.751048 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8fedf0883b4b5ebee7e028a94a503b" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:45.751161 11247 raft_consensus.cc:3037] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:45.751271 14063 raft_consensus.cc:2454] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad18cd54c6340fd92b874157875fa49 in term 2.
W20240501 19:56:45.751509 11313 leader_election.cc:334] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.751657 11577 leader_election.cc:302] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8ad18cd54c6340fd92b874157875fa49; no voters: 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:45.751835 18724 raft_consensus.cc:2781] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:45.751931 18724 raft_consensus.cc:686] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [term 2 LEADER]: Becoming Leader. State: Replica: 8ad18cd54c6340fd92b874157875fa49, State: Running, Role: LEADER
I20240501 19:56:45.752084 18724 consensus_queue.cc:237] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } }
I20240501 19:56:45.753739 11247 raft_consensus.cc:2454] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 2.
I20240501 19:56:45.754062 11313 leader_election.cc:302] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:45.754293 18778 raft_consensus.cc:2781] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:45.754397 18778 raft_consensus.cc:686] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [term 2 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:45.754293 11100 catalog_manager.cc:5484] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 reported cstate change: term changed from 1 to 2, leader changed from 6921cf2e41f64d66842885512da8de6c (127.5.40.7) to 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4). New cstate: current_term: 2 leader_uuid: "8ad18cd54c6340fd92b874157875fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.754523 18778 consensus_queue.cc:237] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } }
I20240501 19:56:45.756943 18793 raft_consensus.cc:482] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.756876 11098 catalog_manager.cc:5484] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 1 to 2, leader changed from 6921cf2e41f64d66842885512da8de6c (127.5.40.7) to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 2 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.757050 18793 raft_consensus.cc:504] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:45.757449 18793 leader_election.cc:288] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:45.758005 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e3882ea4e4dbd97959c03a43de6d9" candidate_uuid: "08711534813d47ba85aaec0771ed5d0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
W20240501 19:56:45.758554 11448 leader_election.cc:334] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.758625 11448 leader_election.cc:302] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f; no voters: 10ee60ead7b94c93ab749ec16a04b0dc, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.758752 18793 raft_consensus.cc:2726] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.770846 18775 raft_consensus.cc:482] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.770982 18775 raft_consensus.cc:504] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:45.771375 18775 leader_election.cc:288] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:45.772495 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da88b92a3fc4f1a89e7eb990557d7d7" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
W20240501 19:56:45.773488 11184 leader_election.cc:334] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.773561 11184 leader_election.cc:302] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc; no voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.773722 18775 raft_consensus.cc:2726] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.779532 18778 raft_consensus.cc:482] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.779664 18778 raft_consensus.cc:504] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:45.780092 18778 leader_election.cc:288] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.780452 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85f5acd65914c2d8f1cc48d668c40eb" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:45.780598 13667 raft_consensus.cc:2454] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 1.
I20240501 19:56:45.781062 11313 leader_election.cc:302] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, ef60ab93b49c4346af4f62d3b98558b8; no voters:
W20240501 19:56:45.781193 11313 leader_election.cc:334] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.781222 18778 raft_consensus.cc:2781] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:45.781265 18778 raft_consensus.cc:482] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:45.781312 18778 raft_consensus.cc:3037] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:45.782150 18778 raft_consensus.cc:504] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:45.782482 18778 leader_election.cc:288] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: Requested vote from peers 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
I20240501 19:56:45.782799 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85f5acd65914c2d8f1cc48d668c40eb" candidate_uuid: "ef60ab93b49c4346af4f62d3b98558b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:45.782925 13667 raft_consensus.cc:3037] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:45.783361 11313 leader_election.cc:334] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.783746 13667 raft_consensus.cc:2454] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef60ab93b49c4346af4f62d3b98558b8 in term 2.
I20240501 19:56:45.784004 11313 leader_election.cc:302] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b, ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c
I20240501 19:56:45.784145 18778 raft_consensus.cc:2781] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:45.784246 18778 raft_consensus.cc:686] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 2 LEADER]: Becoming Leader. State: Replica: ef60ab93b49c4346af4f62d3b98558b8, State: Running, Role: LEADER
I20240501 19:56:45.784446 18778 consensus_queue.cc:237] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } }
I20240501 19:56:45.786396 11098 catalog_manager.cc:5484] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: term changed from 1 to 2, leader changed from 6921cf2e41f64d66842885512da8de6c (127.5.40.7) to ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2). New cstate: current_term: 2 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.827122 18775 raft_consensus.cc:482] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.827250 18775 raft_consensus.cc:504] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:45.827718 18775 leader_election.cc:288] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:45.828088 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e3882ea4e4dbd97959c03a43de6d9" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f" is_pre_election: true
I20240501 19:56:45.828224 11511 raft_consensus.cc:2454] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:45.828506 11181 leader_election.cc:302] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters:
I20240501 19:56:45.828665 18775 raft_consensus.cc:2781] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:45.828722 18775 raft_consensus.cc:482] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.828761 18775 raft_consensus.cc:3037] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:45.828867 11184 leader_election.cc:334] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.829675 18775 raft_consensus.cc:504] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:45.830021 18775 leader_election.cc:288] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: Requested vote from peers 08711534813d47ba85aaec0771ed5d0f (127.5.40.3:45965), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:45.830305 11511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512e3882ea4e4dbd97959c03a43de6d9" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "08711534813d47ba85aaec0771ed5d0f"
I20240501 19:56:45.830420 11511 raft_consensus.cc:3037] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:45.831034 11184 leader_election.cc:334] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.831282 11511 raft_consensus.cc:2454] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 2.
I20240501 19:56:45.831565 11181 leader_election.cc:302] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f, 10ee60ead7b94c93ab749ec16a04b0dc; no voters: 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.831714 18775 raft_consensus.cc:2781] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:45.831800 18775 raft_consensus.cc:686] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:45.832119 18775 consensus_queue.cc:237] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:45.834182 11098 catalog_manager.cc:5484] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 1 to 2, leader changed from 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 2 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.862000 18775 raft_consensus.cc:482] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.862164 18775 raft_consensus.cc:504] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.862651 18775 leader_election.cc:288] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:45.863139 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8078e0cbffa943c98584224d05b0ece0" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:45.863318 11380 raft_consensus.cc:2454] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:45.863670 11183 leader_election.cc:302] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
W20240501 19:56:45.863765 11184 leader_election.cc:334] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.863826 18775 raft_consensus.cc:2781] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:45.863898 18775 raft_consensus.cc:482] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.863942 18775 raft_consensus.cc:3037] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:45.864868 18775 raft_consensus.cc:504] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.865168 18775 leader_election.cc:288] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:45.865556 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8078e0cbffa943c98584224d05b0ece0" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:45.865689 11380 raft_consensus.cc:3037] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:45.866037 11184 leader_election.cc:334] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.866739 11380 raft_consensus.cc:2454] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 2.
I20240501 19:56:45.867012 11183 leader_election.cc:302] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters: 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.867206 18775 raft_consensus.cc:2781] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:45.867275 18775 raft_consensus.cc:686] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:45.867430 18775 consensus_queue.cc:237] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.869422 11098 catalog_manager.cc:5484] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 1 to 2, leader changed from 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 2 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.881858 18777 raft_consensus.cc:482] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.881994 18777 raft_consensus.cc:504] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:45.882407 18777 leader_election.cc:288] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:45.882956 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9c3a48f92e40f19d75f4151bb4c3e3" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:45.883092 14063 raft_consensus.cc:2454] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 1.
W20240501 19:56:45.883221 13606 leader_election.cc:334] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.883383 13603 leader_election.cc:302] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.883546 18777 raft_consensus.cc:2781] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:45.883603 18777 raft_consensus.cc:482] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.883647 18777 raft_consensus.cc:3037] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:45.884485 18777 raft_consensus.cc:504] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:45.884793 18777 leader_election.cc:288] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:45.885138 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9c3a48f92e40f19d75f4151bb4c3e3" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a"
I20240501 19:56:45.885268 14063 raft_consensus.cc:3037] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:45.885650 13606 leader_election.cc:334] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.886194 14063 raft_consensus.cc:2454] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8125d8ad6d0d44caae9c5c90c90df96b in term 2.
I20240501 19:56:45.886459 13603 leader_election.cc:302] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.886652 18777 raft_consensus.cc:2781] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:45.886756 18777 raft_consensus.cc:686] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 LEADER]: Becoming Leader. State: Replica: 8125d8ad6d0d44caae9c5c90c90df96b, State: Running, Role: LEADER
I20240501 19:56:45.886914 18777 consensus_queue.cc:237] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } }
I20240501 19:56:45.889149 11098 catalog_manager.cc:5484] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b reported cstate change: term changed from 1 to 2, leader changed from 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) to 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5). New cstate: current_term: 2 leader_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.898962 18775 raft_consensus.cc:482] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.899101 18775 raft_consensus.cc:504] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.899698 18775 leader_election.cc:288] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
I20240501 19:56:45.899961 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aabbb9dc2440f28a3b6944dea38fe7" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8" is_pre_election: true
I20240501 19:56:45.900105 11380 raft_consensus.cc:2454] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 1.
I20240501 19:56:45.900616 11183 leader_election.cc:302] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters:
W20240501 19:56:45.900784 11184 leader_election.cc:334] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.900805 18775 raft_consensus.cc:2781] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:45.900944 18775 raft_consensus.cc:482] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.901000 18775 raft_consensus.cc:3037] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:45.902323 18775 raft_consensus.cc:504] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.903210 11380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aabbb9dc2440f28a3b6944dea38fe7" candidate_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef60ab93b49c4346af4f62d3b98558b8"
I20240501 19:56:45.903365 11380 raft_consensus.cc:3037] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:45.903582 18775 leader_election.cc:288] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), ef60ab93b49c4346af4f62d3b98558b8 (127.5.40.2:40185)
W20240501 19:56:45.904063 11184 leader_election.cc:334] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.904527 11380 raft_consensus.cc:2454] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ee60ead7b94c93ab749ec16a04b0dc in term 2.
I20240501 19:56:45.904811 11183 leader_election.cc:302] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, ef60ab93b49c4346af4f62d3b98558b8; no voters: 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.904970 18775 raft_consensus.cc:2781] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:45.905061 18775 raft_consensus.cc:686] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 LEADER]: Becoming Leader. State: Replica: 10ee60ead7b94c93ab749ec16a04b0dc, State: Running, Role: LEADER
I20240501 19:56:45.905212 18775 consensus_queue.cc:237] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } }
I20240501 19:56:45.907124 11098 catalog_manager.cc:5484] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc reported cstate change: term changed from 1 to 2, leader changed from 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) to 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1). New cstate: current_term: 2 leader_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.940961 18340 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
W20240501 19:56:45.941112 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:45.941094 18340 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:45.941529 18340 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:45.942431 13996 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:45.942516 13999 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.942576 13999 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.942715 18340 raft_consensus.cc:2726] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:45.974273 18801 raft_consensus.cc:482] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:45.974390 18801 raft_consensus.cc:504] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:45.974773 18801 leader_election.cc:288] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:45.975720 13603 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:45.975808 13606 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:45.975886 13606 leader_election.cc:302] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:45.976013 18801 raft_consensus.cc:2726] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:45.981886 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.004412 11249 raft_consensus.cc:1261] T d7bb9ef27c6e4f248a0e5a4812986f38 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:46.004845 18307 consensus_queue.cc:1035] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:46.008397 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:46.015044 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.038901 18724 raft_consensus.cc:482] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:46.039034 18724 raft_consensus.cc:504] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:46.039395 18724 leader_election.cc:288] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:46.040311 11577 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:46.040424 11580 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.040522 11580 leader_election.cc:302] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:46.040652 18724 raft_consensus.cc:2726] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:46.064117 18340 raft_consensus.cc:482] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:46.064262 18340 raft_consensus.cc:504] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:46.064404 18801 raft_consensus.cc:482] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:46.064532 18801 raft_consensus.cc:504] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:46.064728 18340 leader_election.cc:288] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:46.064935 18801 leader_election.cc:288] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8:34789)
I20240501 19:56:46.065171 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da88b92a3fc4f1a89e7eb990557d7d7" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" is_pre_election: true
I20240501 19:56:46.065326 11249 raft_consensus.cc:2454] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:46.065503 14063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efaba1013424acd949123503dc503b8" candidate_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "533908e517b24f77b2af8b5863e4ae5a" is_pre_election: true
I20240501 19:56:46.065658 13996 leader_election.cc:302] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 533908e517b24f77b2af8b5863e4ae5a; no voters:
I20240501 19:56:46.065842 18340 raft_consensus.cc:2781] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:46.065901 18340 raft_consensus.cc:482] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:46.065939 18340 raft_consensus.cc:3037] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:46.066247 13606 leader_election.cc:334] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.066488 13606 leader_election.cc:302] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 533908e517b24f77b2af8b5863e4ae5a, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:46.066670 18801 raft_consensus.cc:2726] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:46.066982 13999 leader_election.cc:334] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.067020 18340 raft_consensus.cc:504] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:46.067350 18340 leader_election.cc:288] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 election: Requested vote from peers 10ee60ead7b94c93ab749ec16a04b0dc (127.5.40.1:42965), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:46.067510 18340 raft_consensus.cc:482] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:46.067574 18340 raft_consensus.cc:504] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:46.067670 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da88b92a3fc4f1a89e7eb990557d7d7" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "10ee60ead7b94c93ab749ec16a04b0dc"
I20240501 19:56:46.067809 11249 raft_consensus.cc:3037] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:46.067920 18340 leader_election.cc:288] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
W20240501 19:56:46.068209 13999 leader_election.cc:334] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.068328 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efaba1013424acd949123503dc503b8" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" is_pre_election: true
I20240501 19:56:46.068439 13667 raft_consensus.cc:2454] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 1.
I20240501 19:56:46.068747 13996 leader_election.cc:302] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters:
W20240501 19:56:46.068818 13999 leader_election.cc:334] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.068881 18340 raft_consensus.cc:2781] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:46.068889 11249 raft_consensus.cc:2454] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 2.
I20240501 19:56:46.068931 18340 raft_consensus.cc:482] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:46.068965 18340 raft_consensus.cc:3037] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:46.069232 13996 leader_election.cc:302] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ee60ead7b94c93ab749ec16a04b0dc, 533908e517b24f77b2af8b5863e4ae5a; no voters: 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:46.069290 18801 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:46.069365 18801 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:46.069429 18721 raft_consensus.cc:2781] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:46.069526 18721 raft_consensus.cc:686] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 2 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:46.069684 18801 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
I20240501 19:56:46.069665 18721 consensus_queue.cc:237] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } }
I20240501 19:56:46.069998 18340 raft_consensus.cc:504] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:46.070365 18340 leader_election.cc:288] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 election: Requested vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 8125d8ad6d0d44caae9c5c90c90df96b (127.5.40.5:42501)
W20240501 19:56:46.070948 13603 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:46.071527 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efaba1013424acd949123503dc503b8" candidate_uuid: "533908e517b24f77b2af8b5863e4ae5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8125d8ad6d0d44caae9c5c90c90df96b"
I20240501 19:56:46.071636 13667 raft_consensus.cc:3037] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:46.071772 13999 leader_election.cc:334] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.072081 11100 catalog_manager.cc:5484] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 1 to 2, leader changed from 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 2 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:46.072575 13667 raft_consensus.cc:2454] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533908e517b24f77b2af8b5863e4ae5a in term 2.
I20240501 19:56:46.072839 13996 leader_election.cc:302] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a, 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:46.073004 18340 raft_consensus.cc:2781] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:46.073145 18340 raft_consensus.cc:686] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [term 2 LEADER]: Becoming Leader. State: Replica: 533908e517b24f77b2af8b5863e4ae5a, State: Running, Role: LEADER
I20240501 19:56:46.073271 18340 consensus_queue.cc:237] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
W20240501 19:56:46.074523 13606 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.074726 13606 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:46.074874 18801 raft_consensus.cc:2726] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:46.076128 11100 catalog_manager.cc:5484] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: term changed from 1 to 2, leader changed from 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6) to 533908e517b24f77b2af8b5863e4ae5a (127.5.40.8). New cstate: current_term: 2 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:46.183416 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.196293 11249 raft_consensus.cc:1261] T 2d8fedf0883b4b5ebee7e028a94a503b P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:46.196717 18307 consensus_queue.cc:1035] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:46.212268 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.215191 14063 raft_consensus.cc:1261] T ba2b4dda83bc408eaddec5ebc7b734d5 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Refusing update from remote peer 08711534813d47ba85aaec0771ed5d0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:46.215657 18744 consensus_queue.cc:1035] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:56:46.220917 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:46.223098 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:46.259630 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.274616 14063 raft_consensus.cc:1261] T 313fb8dab2644b339cdd22647a8550fc P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Refusing update from remote peer 8ad18cd54c6340fd92b874157875fa49: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:46.275060 18724 consensus_queue.cc:1035] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:46.286037 13667 raft_consensus.cc:1261] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:46.286489 18307 consensus_queue.cc:1035] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 19:56:46.289036 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.289054 11643 raft_consensus.cc:3037] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49 [term 1 LEARNER]: Advancing to term 2
I20240501 19:56:46.290249 11643 raft_consensus.cc:1261] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49 [term 2 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:46.290742 16220 consensus_queue.cc:1035] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: NON_VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:46.291422 11380 raft_consensus.cc:1261] T 8078e0cbffa943c98584224d05b0ece0 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:56:46.291950 18775 consensus_queue.cc:1035] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:46.293023 18307 raft_consensus.cc:1050] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8: attempting to promote NON_VOTER 8ad18cd54c6340fd92b874157875fa49 to VOTER
I20240501 19:56:46.293432 18307 consensus_queue.cc:237] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:46.295213 13667 raft_consensus.cc:1261] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:46.295625 18307 consensus_queue.cc:1035] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.295940 11511 raft_consensus.cc:1261] T 512e3882ea4e4dbd97959c03a43de6d9 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:46.296721 18776 consensus_queue.cc:1035] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:46.297243 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.297708 11643 raft_consensus.cc:1261] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49 [term 2 LEARNER]: Refusing update from remote peer ef60ab93b49c4346af4f62d3b98558b8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:46.298099 18778 consensus_queue.cc:1035] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.300459 16220 raft_consensus.cc:2932] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:46.301342 11380 raft_consensus.cc:1261] T 87aabbb9dc2440f28a3b6944dea38fe7 P ef60ab93b49c4346af4f62d3b98558b8 [term 2 FOLLOWER]: Refusing update from remote peer 10ee60ead7b94c93ab749ec16a04b0dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:46.301693 18775 consensus_queue.cc:1035] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:46.302543 11643 raft_consensus.cc:2932] T b85f5acd65914c2d8f1cc48d668c40eb P 8ad18cd54c6340fd92b874157875fa49 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
I20240501 19:56:46.303157 13667 raft_consensus.cc:2932] T b85f5acd65914c2d8f1cc48d668c40eb P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } } }
W20240501 19:56:46.304579 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.305639 11102 catalog_manager.cc:5484] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 reported cstate change: config changed from index 2 to 4, 8ad18cd54c6340fd92b874157875fa49 (127.5.40.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ef60ab93b49c4346af4f62d3b98558b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:46.343374 14063 raft_consensus.cc:1261] T af9c3a48f92e40f19d75f4151bb4c3e3 P 533908e517b24f77b2af8b5863e4ae5a [term 2 FOLLOWER]: Refusing update from remote peer 8125d8ad6d0d44caae9c5c90c90df96b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:46.343858 18801 consensus_queue.cc:1035] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b [LEADER]: Connected to new peer: Peer: permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:56:46.347220 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:46.494776 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:46.511641 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:46.517308 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.519946 11249 raft_consensus.cc:1261] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:46.520376 18340 consensus_queue.cc:1035] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:46.523563 11511 raft_consensus.cc:3037] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f [term 1 LEARNER]: Advancing to term 2
I20240501 19:56:46.524787 11511 raft_consensus.cc:1261] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f [term 2 LEARNER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:46.525146 18340 consensus_queue.cc:1035] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: NON_VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 19:56:46.526858 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:46.528357 16195 raft_consensus.cc:1050] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a: attempting to promote NON_VOTER 08711534813d47ba85aaec0771ed5d0f to VOTER
I20240501 19:56:46.528705 16195 consensus_queue.cc:237] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:46.530584 11249 raft_consensus.cc:1261] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:46.530947 11511 raft_consensus.cc:1261] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f [term 2 LEARNER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240501 19:56:46.531170 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.531404 18340 consensus_queue.cc:1035] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.531549 18856 consensus_queue.cc:1035] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:46.533272 18340 raft_consensus.cc:2932] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:46.533720 11511 raft_consensus.cc:2932] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:46.535261 11249 raft_consensus.cc:2932] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 10ee60ead7b94c93ab749ec16a04b0dc [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } } }
I20240501 19:56:46.537000 11102 catalog_manager.cc:5484] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a reported cstate change: config changed from index 2 to 4, 08711534813d47ba85aaec0771ed5d0f (127.5.40.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "533908e517b24f77b2af8b5863e4ae5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ee60ead7b94c93ab749ec16a04b0dc" member_type: VOTER last_known_addr { host: "127.5.40.1" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 19:56:46.632952 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.637302 13667 raft_consensus.cc:1261] T 8efaba1013424acd949123503dc503b8 P 8125d8ad6d0d44caae9c5c90c90df96b [term 2 FOLLOWER]: Refusing update from remote peer 533908e517b24f77b2af8b5863e4ae5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:46.637763 18857 consensus_queue.cc:1035] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:56:46.642530 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:46.651046 11092 catalog_manager.cc:4573] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0bc4ce342be94b309a35450d3c59a6ce: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:46.745775 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:46.810663 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:46.845185 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:46.908737 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:46.910276 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:46.911468 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:46.918973 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.005786 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:47.013911 18853 raft_consensus.cc:482] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:47.014065 18853 raft_consensus.cc:504] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:47.014575 18853 leader_election.cc:288] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:47.015717 13603 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:47.015916 13606 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:47.016058 13606 leader_election.cc:302] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:47.016255 18853 raft_consensus.cc:2726] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:47.049018 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.058182 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.059906 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:47.060217 18307 raft_consensus.cc:482] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:47.060315 18307 raft_consensus.cc:504] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:47.060678 18307 leader_election.cc:288] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:47.061645 11313 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:47.061774 11316 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:47.061856 11316 leader_election.cc:302] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:47.061990 18307 raft_consensus.cc:2726] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:47.115846 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.116578 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.122560 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.123015 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.127825 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.128468 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.133986 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.150048 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.258687 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.300827 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.360806 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.433176 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.446977 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:47.467605 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.480469 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:47.500269 18744 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:47.500411 18744 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:47.500811 18744 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:47.501791 11445 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:47.501893 11448 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:47.501986 11448 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:47.502122 18793 raft_consensus.cc:2726] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:56:47.519634 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:47.519899 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:47.550784 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:47.582180 18853 raft_consensus.cc:482] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:47.582298 18853 raft_consensus.cc:504] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:47.582649 18853 leader_election.cc:288] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:47.583770 13606 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:47.584080 13603 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:47.584148 13603 leader_election.cc:302] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:47.584278 18853 raft_consensus.cc:2726] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:47.689646 11092 catalog_manager.cc:4573] TS 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0bc4ce342be94b309a35450d3c59a6ce: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:47.698444 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.717629 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.721347 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.766842 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.775606 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:47.779070 18860 raft_consensus.cc:482] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:47.779194 18860 raft_consensus.cc:504] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:47.779598 18860 leader_election.cc:288] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:47.780462 11577 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:47.780547 11580 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:47.780607 11580 leader_election.cc:302] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:47.780728 18860 raft_consensus.cc:2726] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:47.818413 18853 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:47.818534 18853 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
W20240501 19:56:47.818930 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:47.818979 18853 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:47.819948 13606 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:47.820122 13603 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:47.820188 13603 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:47.820323 18853 raft_consensus.cc:2726] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:47.836823 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:47.864856 18340 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:47.865000 18340 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:47.865392 18340 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:47.866389 13999 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:47.866549 13996 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:47.866632 13996 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:47.866770 18340 raft_consensus.cc:2726] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:47.913213 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:47.947222 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:47.966490 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.018851 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:48.031057 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.066642 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:48.202695 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.399907 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:48.442144 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.603770 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:48.621026 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:48.757999 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.784020 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.796150 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.853185 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.854125 11089 catalog_manager.cc:4573] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): CreateTablet RPC failed for tablet ad5d7cd91f474de6a2ec6369e4776def: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:48.863947 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.929237 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:48.971870 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:49.029911 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:49.031378 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:49.061391 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:49.166188 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:49.174925 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.180684 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:49.254523 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.273075 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.311151 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.315351 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.320071 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:49.396353 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.454943 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:49.456835 18853 raft_consensus.cc:482] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:49.456975 18853 raft_consensus.cc:504] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:49.457449 18853 leader_election.cc:288] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:49.458506 13606 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:49.458602 13603 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:49.458683 13603 leader_election.cc:302] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:49.458810 18853 raft_consensus.cc:2726] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:49.486459 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.524995 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.525332 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.564759 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.566591 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:49.583133 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.589994 11313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240501 19:56:49.617916 11184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 19:56:49.620611 13606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 19:56:49.647826 13996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 19:56:49.655607 11092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240501 19:56:49.657459 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.677371 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.718240 11448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 19:56:49.718448 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:49.730381 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.737114 11580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240501 19:56:49.743371 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.746608 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.782361 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:49.889657 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:56:49.922509 18872 raft_consensus.cc:482] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:49.922634 18872 raft_consensus.cc:504] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:49.923054 18872 leader_election.cc:288] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:49.924134 11580 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:49.924229 11577 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:49.924317 11577 leader_election.cc:302] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:49.924461 18872 raft_consensus.cc:2726] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:49.929150 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:49.934754 11089 catalog_manager.cc:4573] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:49.991281 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:49.999284 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.021850 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.071854 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.072520 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.114045 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.141589 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.155970 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:56:50.242640 18887 raft_consensus.cc:482] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:50.242763 18887 raft_consensus.cc:504] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:50.243201 18887 leader_election.cc:288] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:50.244211 11316 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:50.244308 11313 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:50.244366 11313 leader_election.cc:302] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:50.244486 18887 raft_consensus.cc:2726] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:50.245946 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.249097 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.264832 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.283558 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.288726 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:50.325846 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:56:50.329993 18881 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:50.330137 18881 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:50.330546 18881 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:50.331516 13606 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:50.331851 13603 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:50.331929 13603 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:50.332068 18881 raft_consensus.cc:2726] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:50.349165 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:50.408696 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:50.410576 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.466171 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.488056 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.501051 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:56:50.599725 18881 raft_consensus.cc:482] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:50.599838 18881 raft_consensus.cc:504] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:50.600214 18881 leader_election.cc:288] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:50.601136 13603 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:50.601229 13606 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:50.601315 13606 leader_election.cc:302] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:50.601440 18881 raft_consensus.cc:2726] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:50.655046 18873 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:50.655179 18873 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:50.655609 18873 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:50.656659 11445 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:50.656985 11448 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:50.657061 11448 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:50.657204 18873 raft_consensus.cc:2726] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:56:50.670850 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.722923 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:50.833734 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:50.926606 18871 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:50.926744 18871 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:50.927191 18871 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:50.928346 13996 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:50.928442 13999 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:50.928498 13999 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:50.928649 18871 raft_consensus.cc:2726] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:50.929466 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:50.971240 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:51.230381 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.231017 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.312548 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.337133 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.340826 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:51.347471 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.355199 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:51.355316 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.374913 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:51.414624 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.465095 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:51.517073 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.602901 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:51.674789 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:51.676512 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.734437 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:51.812784 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:51.833988 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:51.837734 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:51.852136 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:51.887629 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:51.893162 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:51.968693 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:51.996387 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.011776 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.055182 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.081573 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:56:52.141705 18881 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:52.141834 18881 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:52.142232 18881 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:52.143252 13603 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:52.143373 13606 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:52.143488 13606 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:52.143640 18881 raft_consensus.cc:2726] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:52.202730 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.219276 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.258920 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.264101 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.279901 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:52.330613 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.359906 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:52.399868 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.420131 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:52.446022 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.466706 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.472179 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.526372 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.537307 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.543538 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.612582 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.665906 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.682253 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.707896 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.726114 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.731308 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.781900 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.811913 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:52.818688 18895 raft_consensus.cc:482] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:52.818805 18895 raft_consensus.cc:504] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:52.819200 18895 leader_election.cc:288] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:52.820142 11313 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:52.820233 11316 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:52.820291 11316 leader_election.cc:302] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:52.820427 18895 raft_consensus.cc:2726] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:52.862046 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.899125 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:52.922026 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:52.960466 11089 catalog_manager.cc:4573] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): CreateTablet RPC failed for tablet ad5d7cd91f474de6a2ec6369e4776def: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:52.967978 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:53.012217 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:53.014046 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:53.330960 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:53.380033 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:53.482031 18881 raft_consensus.cc:482] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:53.482167 18881 raft_consensus.cc:504] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:53.482573 18881 leader_election.cc:288] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:53.483515 13603 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:53.483604 13606 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:53.483680 13606 leader_election.cc:302] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:53.483820 18881 raft_consensus.cc:2726] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:53.489223 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:53.497978 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:53.499980 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:53.573503 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:53.575068 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:53.576222 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:53.603802 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:53.780545 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:56:53.789793 18904 raft_consensus.cc:482] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:53.789945 18904 raft_consensus.cc:504] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:53.790314 18904 leader_election.cc:288] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:53.791468 11580 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:53.791599 11577 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:53.791927 11577 leader_election.cc:302] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:53.792181 18904 raft_consensus.cc:2726] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:53.810021 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:53.922827 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:53.976637 18914 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:53.976768 18914 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:53.977453 18914 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:53.978178 11448 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:53.978281 11445 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:53.978350 11445 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:53.978466 18914 raft_consensus.cc:2726] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:56:53.988773 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:53.996474 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:54.019543 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:54.043530 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:54.068938 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:54.082087 11089 catalog_manager.cc:4573] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:54.083418 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:54.183859 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.215112 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.318347 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.343801 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.346740 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:54.391955 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.395730 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:54.396136 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.510927 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.531392 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:54.546166 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.566030 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.635051 11313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240501 19:56:54.635906 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:54.660574 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.723014 11184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 19:56:54.738507 11577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240501 19:56:54.739066 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.740181 13603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 19:56:54.792735 11448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240501 19:56:54.797521 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.826256 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.826337 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.844334 13996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 19:56:54.844965 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.905120 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.917810 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:54.967933 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.971812 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:54.976224 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:56:54.977613 18922 raft_consensus.cc:482] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:54.977711 18922 raft_consensus.cc:504] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:54.978075 18922 leader_election.cc:288] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:54.978847 13603 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:54.978976 13606 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:54.979033 13606 leader_election.cc:302] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:54.979161 18922 raft_consensus.cc:2726] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:54.998054 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.014204 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:55.026577 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.085103 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.115144 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:55.121721 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.126281 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.192116 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.221616 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.235141 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.273061 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.301012 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.317543 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.452373 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:55.490356 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:55.506714 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:56:55.519888 18899 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:55.520031 18899 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:56:55.520414 18899 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:55.521395 13999 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:55.521512 13996 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:55.521600 13996 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:55.521739 18899 raft_consensus.cc:2726] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:55.568625 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.622905 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:55.644125 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.964282 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:55.970187 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:55.993170 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:56.055810 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.098944 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:56.134603 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.140583 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.162025 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.242242 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.247984 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.394491 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:56.423399 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.432404 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:56:56.525425 18881 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:56.525549 18881 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:56:56.525908 18881 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:56.526923 13606 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:56.527022 13603 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:56.527084 13603 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:56.527215 18881 raft_consensus.cc:2726] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:56.529757 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.554131 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:56.570080 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:56.575309 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.596526 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:56.616441 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:56.617678 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:56.638834 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.733085 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:56.762249 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:56.771713 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:56.819201 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:56.835664 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:56.852509 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:56.970400 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:56.981225 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:57.039038 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.135386 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:57.157191 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.159494 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.248716 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.306080 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.315344 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.337747 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.351869 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.402500 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.458274 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:57.465477 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.476662 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.483630 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.500833 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:57.520766 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.528324 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.540594 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.546284 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.593935 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:57.624473 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.627910 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.660770 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:56:57.750442 18895 raft_consensus.cc:482] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:57.750578 18895 raft_consensus.cc:504] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:56:57.751035 18895 leader_election.cc:288] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:57.752069 11316 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:57.752177 11313 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:57.752238 11313 leader_election.cc:302] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:57.752367 18895 raft_consensus.cc:2726] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:57.763890 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.765053 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.828233 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:57.949339 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:58.026316 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:58.039746 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.069111 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:58.300967 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.372502 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.389389 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.448051 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:58.520913 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.540477 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:58.544009 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.552436 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:58.701792 18936 raft_consensus.cc:482] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:56:58.701923 18936 raft_consensus.cc:504] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:56:58.702288 18936 leader_election.cc:288] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:58.703198 11577 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:58.703294 11580 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:58.703351 11580 leader_election.cc:302] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:58.703495 18936 raft_consensus.cc:2726] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:56:58.753451 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.793985 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:58.851037 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:58.880731 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.913486 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.928212 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:58.930333 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.962666 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:58.981984 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:58.996625 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:59.142715 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.182526 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.183185 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:59.187348 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:59.196267 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:59.218605 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:59.244617 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:56:59.246932 18920 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:56:59.247083 18920 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:56:59.247570 18920 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:56:59.248554 11445 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:59.248641 11448 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:59.248718 11448 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:59.248843 18920 raft_consensus.cc:2726] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:56:59.301513 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.321532 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.332129 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.334664 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.375109 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:56:59.417366 18944 raft_consensus.cc:482] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:59.417482 18944 raft_consensus.cc:504] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:56:59.418090 18944 leader_election.cc:288] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:56:59.418870 13606 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:59.419023 13603 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:56:59.419092 13603 leader_election.cc:302] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:56:59.419217 18944 raft_consensus.cc:2726] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:56:59.421041 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:59.422188 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.539012 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.578248 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:59.720654 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.734625 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.746181 11316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 19:56:59.746806 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.775259 11184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 19:56:59.820426 11577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 19:56:59.822747 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.824563 13606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240501 19:56:59.844885 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.869104 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:56:59.875977 13996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 19:56:59.876562 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.912473 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.939239 11448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 19:56:59.966949 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:59.976811 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.977062 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:56:59.984594 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:59.997939 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.008927 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.101763 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.138005 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.176846 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.210767 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.242116 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.255393 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:57:00.290539 18943 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:57:00.290688 18943 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:57:00.291121 18943 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:57:00.292068 13996 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:00.292198 13999 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:57:00.292258 13999 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:00.292372 18943 raft_consensus.cc:2726] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:00.342324 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.362433 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:00.424490 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.435459 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.486106 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.488253 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:00.524217 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:00.823985 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:00.900864 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:00.921604 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:00.990062 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.001411 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.001966 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.030380 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:01.081144 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:01.128826 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.163722 11089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240501 19:57:01.164402 11089 catalog_manager.cc:4573] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): CreateTablet RPC failed for tablet ad5d7cd91f474de6a2ec6369e4776def: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:01.386780 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:01.388202 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.409500 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.421831 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:01.434993 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.492580 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.543797 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.546378 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.551884 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:01.560712 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:01.578444 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:01.638217 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:01.663362 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:01.687347 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:01.687767 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.745386 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.750898 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:01.772123 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:01.807389 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:01.910118 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:01.950667 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.013653 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.024761 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.045356 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:02.053419 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.183012 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:57:02.189934 18956 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:57:02.190069 18956 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:57:02.190472 18956 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:57:02.192121 13603 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:02.192294 13606 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:57:02.192435 13606 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:02.192616 18956 raft_consensus.cc:2726] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:02.227300 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.237579 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:02.238503 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.256971 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.306550 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.314725 11089 catalog_manager.cc:4573] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9dad5cfa9b7a4a808eb9d0bb81f2a2c6: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:02.317931 18895 raft_consensus.cc:482] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:02.318048 18895 raft_consensus.cc:504] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:57:02.318436 18895 leader_election.cc:288] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:57:02.319372 11316 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:02.319532 11313 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:02.319597 11313 leader_election.cc:302] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:02.319787 18895 raft_consensus.cc:2726] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:57:02.325906 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:02.362257 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.390857 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.419485 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.467149 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:57:02.481287 18956 raft_consensus.cc:482] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:57:02.481411 18956 raft_consensus.cc:504] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:57:02.481794 18956 leader_election.cc:288] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:57:02.482618 13603 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:02.482698 13606 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:57:02.482756 13606 leader_election.cc:302] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:02.482880 18956 raft_consensus.cc:2726] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:02.514437 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.528023 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:02.573668 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.593149 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.686643 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.695248 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:02.796065 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.814023 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:02.818295 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.845268 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.851475 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.854043 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.964587 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:02.983855 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:03.093762 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:03.215585 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:03.287884 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:03.388180 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:03.396667 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:03.485790 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:57:03.488246 18943 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:57:03.488365 18943 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:57:03.488732 18943 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:57:03.489620 13999 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:03.489706 13996 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:03.489766 13996 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:03.489904 18943 raft_consensus.cc:2726] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:03.504892 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:03.586270 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:03.591192 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:03.778973 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:03.805860 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:03.814788 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:03.864959 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:03.872226 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:03.893371 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:03.953037 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.008212 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.028832 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:04.037315 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:04.052532 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:04.054942 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.055531 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.060457 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:04.166057 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:04.203326 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:04.254352 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:04.317497 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:04.361989 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.383209 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:04.429039 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:04.430912 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.433087 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.488147 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:04.515559 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.589412 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.603953 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.628206 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.661312 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:04.691988 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.693190 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.768863 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:04.819696 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.831554 11316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240501 19:57:04.832166 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.843420 11577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 19:57:04.868690 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.879616 13996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 19:57:04.885560 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.903607 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:04.919983 13606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240501 19:57:04.928901 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 19:57:04.978880 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:04.983861 11445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 19:57:05.055965 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.099949 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:05.115033 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.141584 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.165753 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.210450 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.230188 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.294071 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.295315 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:05.302270 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.322474 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.374013 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.405153 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:05.462792 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.580091 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:05.686400 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.704327 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.787572 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:05.831060 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:05.854452 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:05.868477 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.002081 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.020329 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.097671 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:06.198379 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:06.220804 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.363734 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.401448 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:06.407300 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:06.419853 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:06.428385 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:06.430233 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.494449 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:06.511355 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:06.643301 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.650403 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.669903 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.678979 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:57:06.680294 18987 raft_consensus.cc:482] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:06.680414 18987 raft_consensus.cc:504] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } }
I20240501 19:57:06.680861 18987 leader_election.cc:288] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:57:06.681738 13603 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:06.681829 13606 leader_election.cc:334] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:57:06.681888 13606 leader_election.cc:302] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:06.682013 18987 raft_consensus.cc:2726] T ad5d7cd91f474de6a2ec6369e4776def P 8125d8ad6d0d44caae9c5c90c90df96b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:57:06.771613 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:06.776646 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:06.792268 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:06.848456 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:06.880970 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:06.955564 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:06.962975 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:06.997836 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:07.008740 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:57:07.013109 18991 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:57:07.013238 18991 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:57:07.013646 18991 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:57:07.014511 11448 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:07.014600 11445 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:07.014683 11445 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:07.014815 18991 raft_consensus.cc:2726] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:07.066063 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:07.102190 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.163273 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.195029 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.253630 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:07.287415 18976 raft_consensus.cc:482] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:07.287577 18976 raft_consensus.cc:504] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "ef60ab93b49c4346af4f62d3b98558b8" member_type: VOTER last_known_addr { host: "127.5.40.2" port: 40185 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } }
I20240501 19:57:07.287971 18976 leader_election.cc:288] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:57:07.288947 11316 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:07.289036 11313 leader_election.cc:334] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:07.289094 11313 leader_election.cc:302] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef60ab93b49c4346af4f62d3b98558b8; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:07.289206 18976 raft_consensus.cc:2726] T b73565839e3342e785237ab4ee96c5c9 P ef60ab93b49c4346af4f62d3b98558b8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:57:07.289228 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.343791 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.356827 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.407754 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:07.457758 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.464989 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.484812 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.487227 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:07.522814 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.541146 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.588430 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.716017 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.733448 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.778069 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.812692 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:07.818091 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.824106 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.832603 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.839735 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.888927 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:07.940188 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:08.009505 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:08.046563 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.126734 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.218005 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.263810 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:08.298755 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.344882 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.483615 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:08.490608 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:08.508616 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.612475 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:57:08.629105 18996 raft_consensus.cc:482] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:57:08.629256 18996 raft_consensus.cc:504] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "8ad18cd54c6340fd92b874157875fa49" member_type: VOTER last_known_addr { host: "127.5.40.4" port: 42041 } attrs { promote: false } }
I20240501 19:57:08.629686 18996 leader_election.cc:288] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:57:08.630723 11580 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:08.630802 11577 leader_election.cc:334] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:08.630849 11577 leader_election.cc:302] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ad18cd54c6340fd92b874157875fa49; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:08.630980 18996 raft_consensus.cc:2726] T 7c0ec896817246b19f753f94ff3f7af5 P 8ad18cd54c6340fd92b874157875fa49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:08.649849 18987 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:57:08.649999 18987 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:57:08.650429 18987 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:57:08.651490 13606 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:08.651643 13603 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:08.651719 13603 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:08.651825 18987 raft_consensus.cc:2726] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:08.657955 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.709513 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:08.714257 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.859776 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.878015 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:08.888967 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:08.921517 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:08.932142 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:08.952962 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:08.974848 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:09.052181 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:09.121037 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.162546 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.166877 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:09.202442 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:09.210213 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:09.295150 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:09.311723 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.362901 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:09.364614 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.432588 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:09.436827 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.460587 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.479496 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:09.540333 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.591727 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:09.638144 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.666733 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.784626 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.820919 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.851383 11577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 19:57:09.854506 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:09.885561 11313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 19:57:09.898193 13996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 19:57:09.921098 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:09.958339 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 19:57:09.960384 13606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240501 19:57:09.960423 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:09.972340 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.014330 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.037559 11445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 19:57:10.041249 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.087625 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.088145 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.167337 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.201215 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:10.249439 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.260139 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.265357 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.271730 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.339596 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.411149 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:10.465315 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.477473 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.554880 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.555800 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:10.572688 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.641098 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.756721 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:10.808492 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:10.815272 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.820165 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.825266 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:10.870901 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:10.916743 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:57:11.001686 18987 raft_consensus.cc:482] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:57:11.001818 18987 raft_consensus.cc:504] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:57:11.002179 18987 leader_election.cc:288] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:57:11.003147 13603 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:11.003258 13606 leader_election.cc:334] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:57:11.003327 13606 leader_election.cc:302] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:11.003489 18987 raft_consensus.cc:2726] T 2d4cf91c71044cf6a37528e8c23954c0 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:11.067826 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:57:11.068126 18987 raft_consensus.cc:482] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:57:11.068248 18987 raft_consensus.cc:504] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8125d8ad6d0d44caae9c5c90c90df96b" member_type: VOTER last_known_addr { host: "127.5.40.5" port: 42501 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } }
I20240501 19:57:11.068675 18987 leader_election.cc:288] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243), 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331)
W20240501 19:57:11.069926 13606 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:11.070197 13603 leader_election.cc:334] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:11.070264 13603 leader_election.cc:302] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8125d8ad6d0d44caae9c5c90c90df96b; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:11.070397 18987 raft_consensus.cc:2726] T 7677fcc04dc745cc9657d053fae31c36 P 8125d8ad6d0d44caae9c5c90c90df96b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:11.082785 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:11.117614 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:11.207934 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:11.230932 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:11.244295 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:11.312378 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:11.338034 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:57:11.415603 18981 raft_consensus.cc:482] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96fc18dbb5fd41afb9bebf805e8d33a7)
I20240501 19:57:11.415743 18981 raft_consensus.cc:504] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "533908e517b24f77b2af8b5863e4ae5a" member_type: VOTER last_known_addr { host: "127.5.40.8" port: 34789 } } peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } attrs { promote: false } }
I20240501 19:57:11.416193 18981 leader_election.cc:288] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:57:11.417202 13996 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:11.417304 13999 leader_election.cc:334] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
I20240501 19:57:11.417364 13999 leader_election.cc:302] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 533908e517b24f77b2af8b5863e4ae5a; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:11.417510 18981 raft_consensus.cc:2726] T 1fc692e320d94825964040105a3175cd P 533908e517b24f77b2af8b5863e4ae5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:11.424971 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:11.510845 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:11.549497 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:11.660007 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:11.698012 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:11.721657 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:11.728544 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:11.730144 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:11.730497 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:11.758533 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:11.802495 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:11.804337 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:11.849871 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:11.856701 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:11.878595 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:11.921888 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:11.970117 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:12.023857 11184 consensus_peers.cc:485] T e96e8e7c66584b1198bea464e5a1a4d8 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.076776 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:12.099162 11448 consensus_peers.cc:485] T 2c86f45702e94c0e898e4e81148f2098 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.120308 11316 consensus_peers.cc:485] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.135385 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:12.144366 11445 consensus_peers.cc:485] T 00b7dd445e2846d5a0e1277e160d46bc P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.369987 11313 consensus_peers.cc:485] T 89c074e9edf94d8a8e422d972f0e6982 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:57:12.387502 19012 raft_consensus.cc:482] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6921cf2e41f64d66842885512da8de6c)
I20240501 19:57:12.387651 19012 raft_consensus.cc:504] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96fc18dbb5fd41afb9bebf805e8d33a7" member_type: VOTER last_known_addr { host: "127.5.40.6" port: 46331 } } peers { permanent_uuid: "6921cf2e41f64d66842885512da8de6c" member_type: VOTER last_known_addr { host: "127.5.40.7" port: 36243 } } peers { permanent_uuid: "08711534813d47ba85aaec0771ed5d0f" member_type: VOTER last_known_addr { host: "127.5.40.3" port: 45965 } attrs { promote: false } }
I20240501 19:57:12.388064 19012 leader_election.cc:288] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331), 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243)
W20240501 19:57:12.389147 11448 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:12.389289 11445 leader_election.cc:334] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
I20240501 19:57:12.389343 11445 leader_election.cc:302] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08711534813d47ba85aaec0771ed5d0f; no voters: 6921cf2e41f64d66842885512da8de6c, 96fc18dbb5fd41afb9bebf805e8d33a7
I20240501 19:57:12.389477 19012 raft_consensus.cc:2726] T dcb07fd6b982437195ff42e5bf0c48be P 08711534813d47ba85aaec0771ed5d0f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:57:12.399071 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:12.404064 13999 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.435396 11577 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.498935 11448 consensus_peers.cc:485] T 04932055883c412f944e839982815ab6 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.564740 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:12.575378 11184 consensus_peers.cc:485] T 200886108633404e917f52c7db2a18d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.578536 11445 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.578997 11580 consensus_peers.cc:485] T 6f2498588acd4be0a23017da1b713182 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.656179 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:12.677093 13603 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.688462 11181 consensus_peers.cc:485] T 1c07d9fc05254a6399afcc54b3efb8ac P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.714313 11577 consensus_peers.cc:485] T d505c415daa54005b8ed22bdc11ce135 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.749800 11184 consensus_peers.cc:485] T 512e3882ea4e4dbd97959c03a43de6d9 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.791034 11313 consensus_peers.cc:485] T 8f3176bda2ad4b9986cda48e4a03ce49 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.865397 11316 consensus_peers.cc:485] T cebd15701ee44b11a2f8804740fb4299 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.917927 13996 consensus_peers.cc:485] T 72b93db52ae64bfbb274dbfc5d27eb86 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.929383 11577 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:12.951131 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:12.987289 11580 consensus_peers.cc:485] T 69facf86bba24f209c06afafc1ba8f7b P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.057292 11577 consensus_peers.cc:485] T 1772073172764f8db95c597f8da14f12 P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:13.110231 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:13.115981 13606 consensus_peers.cc:485] T fc5eb3e030774199836cff2a657e8d56 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.153565 13606 consensus_peers.cc:485] T 19d0d6aaa47e4f7eaa80ec5e0207d51d P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.215620 11445 consensus_peers.cc:485] T eff09e5884f44e18a66583e8e0e73679 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.233507 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:13.439229 11184 consensus_peers.cc:485] T 87aabbb9dc2440f28a3b6944dea38fe7 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.475843 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:13.496932 11448 consensus_peers.cc:485] T 33be1f7276c74d49b42c8761e3e4cd4e P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.537693 11184 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:13.557684 11577 consensus_peers.cc:485] T 313fb8dab2644b339cdd22647a8550fc P 8ad18cd54c6340fd92b874157875fa49 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.564556 13606 consensus_peers.cc:485] T af9c3a48f92e40f19d75f4151bb4c3e3 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.632865 11580 consensus_peers.cc:485] T 6caecf09f4114855bddc470eb96bd5af P 8ad18cd54c6340fd92b874157875fa49 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.716270 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:13.761898 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:13.804198 11313 consensus_peers.cc:485] T 2d8fedf0883b4b5ebee7e028a94a503b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.866511 11313 consensus_peers.cc:485] T d7bb9ef27c6e4f248a0e5a4812986f38 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:13.938849 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:13.975275 13996 consensus_peers.cc:485] T 4acffebbd9284c6289361d23cea16b3e P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:13.999310 13996 consensus_peers.cc:485] T 5657f0c0f383445f9c63d6afffbe5bd6 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:14.077203 11316 consensus_peers.cc:485] T 6326c660e6cb4eb289634a68fc50c4af P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:14.142239 13603 consensus_peers.cc:485] T 22e166cb5c864fd993c67c5c08cab1d0 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:14.151064 11184 consensus_peers.cc:485] T 8078e0cbffa943c98584224d05b0ece0 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:14.151731 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:14.164466 11316 consensus_peers.cc:485] T 61cadb3ab334435eaebe99fc3c9e496b P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:14.217284 11181 consensus_peers.cc:485] T c3b35dab7eec4b73b208f687dad46ba2 P 10ee60ead7b94c93ab749ec16a04b0dc -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:14.230507 11448 consensus_peers.cc:462] T 1fc7da07b6c5414bb305264962d0f218 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:14.231966 13603 consensus_peers.cc:485] T 811d018fde314227a378f2d530e0a924 P 8125d8ad6d0d44caae9c5c90c90df96b -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:14.274804 13999 consensus_peers.cc:485] T 8efaba1013424acd949123503dc503b8 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:14.314249 11448 consensus_peers.cc:462] T 6a5f03eecaec4b929723c8f02a185606 P 08711534813d47ba85aaec0771ed5d0f -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111)
W20240501 19:57:14.314807 11313 consensus_peers.cc:485] T b85f5acd65914c2d8f1cc48d668c40eb P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:14.380921 13996 consensus_peers.cc:485] T 33ae450510f54c7baca85259ef97bfa1 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
F20240501 19:57:14.435271 14113 rebalancer_tool-test.cc:1107] Check failed: _s.ok() Bad status: Timed out: Timed out waiting for Table Creation
*** Check failure stack trace: ***
*** Aborted at 1714593434 (unix time) try "date -d @1714593434" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000014a0) received by PID 5280 (TID 0x7f0d6ce5a700) from PID 5280; stack trace: ***
W20240501 19:57:14.439626 11313 consensus_peers.cc:485] T 01a10926bce14cf1bd12f21287e08c89 P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
@ 0x7f0d82e46980 (unknown) at ??:0
@ 0x7f0d8041cfb7 gsignal at ??:0
@ 0x7f0d8041e921 abort at ??:0
@ 0x7f0d8165cdcd google::LogMessage::Fail() at ??:0
@ 0x7f0d81660b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0d8165c7cc google::LogMessage::Flush() at ??:0
@ 0x7f0d8165df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x563eafbdde11 _ZZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvENKUlvE3_clEv at ??:0
@ 0x563eafbf756e _ZSt13__invoke_implIvZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
@ 0x563eafbf62db _ZSt8__invokeIZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ at ??:0
@ 0x563eafbfdcdc _ZNSt6thread8_InvokerISt5tupleIJZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x563eafbfdb65 _ZNSt6thread8_InvokerISt5tupleIJZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_EEEclEv at ??:0
@ 0x563eafbfda30 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_EEEEE6_M_runEv at ??:0
@ 0x7f0d80e426df (unknown) at ??:0
@ 0x7f0d82e3b6db start_thread at ??:0
@ 0x7f0d804ff71f clone at ??:0
W20240501 19:57:14.497108 11089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:14.498044 11089 catalog_manager.cc:4573] TS 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): CreateTablet RPC failed for tablet ad5d7cd91f474de6a2ec6369e4776def: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)
W20240501 19:57:14.498122 11089 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet ad5d7cd91f474de6a2ec6369e4776def on TS 6921cf2e41f64d66842885512da8de6c
W20240501 19:57:14.505019 13999 consensus_peers.cc:485] T 0da88b92a3fc4f1a89e7eb990557d7d7 P 533908e517b24f77b2af8b5863e4ae5a -> Peer 96fc18dbb5fd41afb9bebf805e8d33a7 (127.5.40.6:46331): Couldn't send request to peer 96fc18dbb5fd41afb9bebf805e8d33a7. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.6:46331: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:14.547116 11445 consensus_peers.cc:485] T ba2b4dda83bc408eaddec5ebc7b734d5 P 08711534813d47ba85aaec0771ed5d0f -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Couldn't send request to peer 6921cf2e41f64d66842885512da8de6c. Status: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:14.585783 11313 consensus_peers.cc:462] T e6c1d3814e364e718449c3e4379b31ed P ef60ab93b49c4346af4f62d3b98558b8 -> Peer 6921cf2e41f64d66842885512da8de6c (127.5.40.7:36243): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.40.7:36243: connect: Connection refused (error 111)